[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:12 dist-test-slave-05m0 krb5kdc[1004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:12 dist-test-slave-05m0 krb5kdc[1004](info): set up 2 sockets
Sep 25 00:48:12 dist-test-slave-05m0 krb5kdc[1004](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 00:48:13.757850 997 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:13 dist-test-slave-05m0 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:13.881166 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46163
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:13.989054 1017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:13.989322 1017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:13.989364 1017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:13.993376 1017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:13.993462 1017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:13.993485 1017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:13.993525 1017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:13.993543 1017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:13.998921 1017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:14.000046 1017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:14.001139 1017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:14.010593 1023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:14.010742 1022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:14.010624 1025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:14.010768 1017 server_base.cc:1034] running on GCE node
I20260925 00:48:14.011431 1017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:14.012507 1017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:14.013664 1017 hybrid_clock.cc:648] HybridClock initialized: now 1790297294013652 us; error 25 us; skew 500 ppm
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:14.016285 1017 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:14.017083 1017 webserver.cc:460] Webserver started at http://127.0.249.126:38099/ using document root <none> and password file <none>
I20260925 00:48:14.017649 1017 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:14.017709 1017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:14.017866 1017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:14.019755 1017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "7d103554157e48ebb58800dd0cfe50bc"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "d430108cccc15bb1c3f56669455b1800"
server_key_iv: "ef2c0819257f3051675f7b49cfcad6a6"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.020253 1017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "7d103554157e48ebb58800dd0cfe50bc"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "d430108cccc15bb1c3f56669455b1800"
server_key_iv: "ef2c0819257f3051675f7b49cfcad6a6"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.023787 1017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:48:14.026106 1032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.027351 1017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:14.027486 1017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "7d103554157e48ebb58800dd0cfe50bc"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "d430108cccc15bb1c3f56669455b1800"
server_key_iv: "ef2c0819257f3051675f7b49cfcad6a6"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.027594 1017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:14.040589 1017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:14.043517 1017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:14.043716 1017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:14.050740 1017 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:46163
I20260925 00:48:14.050755 1084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:46163 every 8 connection(s)
I20260925 00:48:14.051644 1017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:14.054369 1085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:14.057317 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 1017
I20260925 00:48:14.057498 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:14.058707 997 external_mini_cluster.cc:1428] Setting key fe1a3aa6e6eb719be9df4c436f71322a
I20260925 00:48:14.059789 1085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc: Bootstrap starting.
I20260925 00:48:14.062390 1085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:14.063241 1085 log.cc:824] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:14.064963 1085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc: No bootstrap required, opened a new log
I20260925 00:48:14.068994 1085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d103554157e48ebb58800dd0cfe50bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46163 } }
I20260925 00:48:14.069195 1085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:14.069252 1085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d103554157e48ebb58800dd0cfe50bc, State: Initialized, Role: FOLLOWER
I20260925 00:48:14.069727 1085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d103554157e48ebb58800dd0cfe50bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46163 } }
I20260925 00:48:14.069875 1085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:14.069950 1085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:14.070047 1085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:14.070863 1085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d103554157e48ebb58800dd0cfe50bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46163 } }
I20260925 00:48:14.071187 1085 leader_election.cc:302] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d103554157e48ebb58800dd0cfe50bc; no voters:
I20260925 00:48:14.071497 1085 leader_election.cc:288] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:14.071560 1090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:14.071748 1090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [term 1 LEADER]: Becoming Leader. State: Replica: 7d103554157e48ebb58800dd0cfe50bc, State: Running, Role: LEADER
I20260925 00:48:14.072021 1090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d103554157e48ebb58800dd0cfe50bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46163 } }
I20260925 00:48:14.072679 1085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:14.073906 1092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d103554157e48ebb58800dd0cfe50bc. Latest consensus state: current_term: 1 leader_uuid: "7d103554157e48ebb58800dd0cfe50bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d103554157e48ebb58800dd0cfe50bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46163 } } }
I20260925 00:48:14.074028 1092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:14.074371 1091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d103554157e48ebb58800dd0cfe50bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d103554157e48ebb58800dd0cfe50bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46163 } } }
I20260925 00:48:14.074461 1091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc [sys.catalog]: This master's current role is: LEADER
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
W20260925 00:48:14.077726 1106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:48:14.077801 1106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:48:14.077858 1099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:14.078684 1099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:14.082984 1087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:14.061716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.0.249.126:46163)
0925 00:48:14.062110 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:14.062124 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:14.063228 (+ 1104us) server_negotiation.cc:407] Connection header received
0925 00:48:14.067114 (+ 3886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:14.067129 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:14.067408 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:14.067726 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:14.069887 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.070984 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.072507 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.073123 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.073283 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:14.077611 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:14.077616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:14.077628 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:14.077652 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:14.079439 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.079963 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.079968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.079972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.080074 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.080410 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.080413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.080428 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.080781 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:14.080986 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:14.081653 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 00:48:14.081952 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":247,"thread_start_us":126,"threads_started":1}
I20260925 00:48:14.085054 1099 catalog_manager.cc:1347] Generated new cluster ID: 0f84b097f6df44bba2b4f1b70bb16f94
I20260925 00:48:14.085114 1099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:14.116262 1099 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:14.117584 1099 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:14.126788 1099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d103554157e48ebb58800dd0cfe50bc: Generated new TSK 0
I20260925 00:48:14.127609 1099 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:48:14.157980 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46163
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:14.267964 1111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:14.268215 1111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:14.268291 1111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:14.272243 1111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:14.272301 1111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:14.272415 1111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:14.277989 1111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:14.279009 1111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:14.280035 1111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:14.289954 1116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:14.289944 1119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:14.290158 1111 server_base.cc:1034] running on GCE node
W20260925 00:48:14.289944 1117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:14.290591 1111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:14.291253 1111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:14.292438 1111 hybrid_clock.cc:648] HybridClock initialized: now 1790297294292403 us; error 45 us; skew 500 ppm
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:14.295436 1111 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:14.296357 1111 webserver.cc:460] Webserver started at http://127.0.249.65:35021/ using document root <none> and password file <none>
I20260925 00:48:14.296880 1111 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:14.296940 1111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:14.297101 1111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:14.299126 1111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "e4832d13506240e9bbb9cf4a575b130e"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "5585896d3883c4171b791bca07209d72"
server_key_iv: "18528065a4e30cf6d91b6882f25be1bb"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.299656 1111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "e4832d13506240e9bbb9cf4a575b130e"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "5585896d3883c4171b791bca07209d72"
server_key_iv: "18528065a4e30cf6d91b6882f25be1bb"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.303359 1111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 00:48:14.305799 1126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.307071 1111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:14.307207 1111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "e4832d13506240e9bbb9cf4a575b130e"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "5585896d3883c4171b791bca07209d72"
server_key_iv: "18528065a4e30cf6d91b6882f25be1bb"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.307315 1111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:14.320091 1111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:14.323220 1111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:14.323419 1111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:14.323976 1111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:14.324906 1111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:14.324977 1111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.325039 1111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:14.325091 1111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.334388 1111 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:33161
I20260925 00:48:14.334408 1239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:33161 every 8 connection(s)
I20260925 00:48:14.335247 1111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:14.344007 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 1111
I20260925 00:48:14.344136 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:14.344383 997 external_mini_cluster.cc:1428] Setting key 7fafa34712a9ee3d315331e02d0ab758
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:14.349159 1087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:14.336994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:59055 (local address 127.0.249.126:46163)
0925 00:48:14.337132 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:14.337136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:14.337949 (+ 813us) server_negotiation.cc:407] Connection header received
0925 00:48:14.339398 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:14.339401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:14.339457 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:14.339564 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:14.340711 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.341247 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.342454 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.342700 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.342811 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:14.345591 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:14.345594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:14.345596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:14.345619 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:14.347145 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.347653 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.347661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.347662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.347735 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.348083 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.348090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.348091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.348256 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:14.348358 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:14.348857 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 00:48:14.348980 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:48:14.350124 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:14.337366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46163 (local address 127.0.249.65:59055)
0925 00:48:14.337802 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:14.337844 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:14.339179 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:14.339746 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:14.339753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:14.340104 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:14.340562 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:14.340577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.341376 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.341386 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:14.342324 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:14.342333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.342859 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.342863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:14.343032 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:14.343583 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:14.343599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:14.345368 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:14.347262 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:14.347268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:14.347283 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:14.347543 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:14.347833 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:14.347836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:14.347837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:14.347973 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:14.348369 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:14.348377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:14.348625 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 00:48:14.348802 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 00:48:14.349116 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260925 00:48:14.351387 1240 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46163
I20260925 00:48:14.351660 1240 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:14.352290 1240 heartbeater.cc:511] Master 127.0.249.126:46163 requested a full tablet report, sending...
I20260925 00:48:14.353905 1049 ts_manager.cc:194] Registered new tserver with Master: e4832d13506240e9bbb9cf4a575b130e (127.0.249.65:33161)
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:48:14.355160 1049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:59055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:14.371865 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46163
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:14.479598 1245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:14.479854 1245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:14.479897 1245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:14.483832 1245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:14.483891 1245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:14.483980 1245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:14.488998 1245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:14.490007 1245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:14.490978 1245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:48:14.500401 1245 server_base.cc:1034] running on GCE node
W20260925 00:48:14.500353 1251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:14.500382 1250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:14.500353 1253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:14.500968 1245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:14.501682 1245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:14.502875 1245 hybrid_clock.cc:648] HybridClock initialized: now 1790297294502862 us; error 28 us; skew 500 ppm
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:14.505685 1245 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:14.506477 1245 webserver.cc:460] Webserver started at http://127.0.249.66:44359/ using document root <none> and password file <none>
I20260925 00:48:14.507040 1245 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:14.507100 1245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:14.507261 1245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:14.509656 1245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "7a559998e9a34436932627dc4ce1c3b8"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "97c4cfe144fae9bd0b7f79d6c8f1213c"
server_key_iv: "2ee1c93dbea1871cbfddd536c08690bd"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.510296 1245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "7a559998e9a34436932627dc4ce1c3b8"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "97c4cfe144fae9bd0b7f79d6c8f1213c"
server_key_iv: "2ee1c93dbea1871cbfddd536c08690bd"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.514535 1245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 00:48:14.516816 1260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.518185 1245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:14.518301 1245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "7a559998e9a34436932627dc4ce1c3b8"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "97c4cfe144fae9bd0b7f79d6c8f1213c"
server_key_iv: "2ee1c93dbea1871cbfddd536c08690bd"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.518414 1245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:14.537587 1245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:14.540782 1245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:14.540982 1245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:14.541636 1245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:14.542440 1245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:14.542495 1245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.542562 1245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:14.542604 1245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.551949 1245 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:39371
I20260925 00:48:14.551988 1373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:39371 every 8 connection(s)
I20260925 00:48:14.552814 1245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:14.557749 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 1245
I20260925 00:48:14.557849 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:14.558115 997 external_mini_cluster.cc:1428] Setting key bdeee5cb6ed0c397215553fce2db0b16
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:14.567526 1087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:14.554526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:47243 (local address 127.0.249.126:46163)
0925 00:48:14.554658 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:14.554662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:14.555401 (+ 739us) server_negotiation.cc:407] Connection header received
0925 00:48:14.556718 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:14.556721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:14.556782 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:14.556858 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:14.558183 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.558876 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.560720 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.560935 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.561048 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:14.563535 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:14.563539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:14.563542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:14.563572 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:14.565384 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.565882 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.565885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.565887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.565982 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.566384 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.566390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.566394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.566625 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:14.566734 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:14.567188 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 00:48:14.567328 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:48:14.568244 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:14.554841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46163 (local address 127.0.249.66:47243)
0925 00:48:14.555255 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:14.555295 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:14.556498 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:14.557017 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:14.557023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:14.557360 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:14.558031 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:14.558049 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.559131 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.559134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:14.560586 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:14.560600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.561089 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.561095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:14.561323 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:14.561973 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:14.561996 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:14.563370 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:14.565525 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:14.565529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:14.565540 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:14.565783 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:14.566079 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:14.566081 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:14.566083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:14.566280 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:14.566730 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:14.566734 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:14.566944 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 00:48:14.567099 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 00:48:14.567335 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260925 00:48:14.569368 1374 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46163
I20260925 00:48:14.569671 1374 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:14.570173 1374 heartbeater.cc:511] Master 127.0.249.126:46163 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
I20260925 00:48:14.571293 1049 ts_manager.cc:194] Registered new tserver with Master: 7a559998e9a34436932627dc4ce1c3b8 (127.0.249.66:39371)
I20260925 00:48:14.571854 1049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:47243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:14.586478 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46163
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:14.695874 1379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:14.696139 1379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:14.696183 1379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:14.700453 1379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:14.700523 1379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:14.700592 1379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:14.705991 1379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:14.707029 1379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:14.708099 1379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:14.717620 1384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:14.717636 1387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:14.717645 1385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:14.718005 1379 server_base.cc:1034] running on GCE node
I20260925 00:48:14.718426 1379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:14.719062 1379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:14.720252 1379 hybrid_clock.cc:648] HybridClock initialized: now 1790297294720236 us; error 26 us; skew 500 ppm
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:14.723222 1379 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:14.724126 1379 webserver.cc:460] Webserver started at http://127.0.249.67:36849/ using document root <none> and password file <none>
I20260925 00:48:14.724666 1379 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:14.724725 1379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:14.724908 1379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:14.727126 1379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "69f38db080bf45d2809db2ab93c88471"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "b3b6eb49656b65e0e49e463d139ef470"
server_key_iv: "a7c237778287b3dcd9c2357b790d219d"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.727723 1379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "69f38db080bf45d2809db2ab93c88471"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "b3b6eb49656b65e0e49e463d139ef470"
server_key_iv: "a7c237778287b3dcd9c2357b790d219d"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.731799 1379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:48:14.734359 1394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.735659 1379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:14.735788 1379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "69f38db080bf45d2809db2ab93c88471"
format_stamp: "Formatted at 2026-09-25 00:48:14 on dist-test-slave-05m0"
server_key: "b3b6eb49656b65e0e49e463d139ef470"
server_key_iv: "a7c237778287b3dcd9c2357b790d219d"
server_key_version: "encryptionkey@0"
I20260925 00:48:14.735900 1379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:14.750061 1379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:14.753552 1379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:14.753762 1379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:14.754386 1379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:14.755198 1379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:14.755244 1379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.755309 1379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:14.755352 1379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:14.764837 1379 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:44599
I20260925 00:48:14.764864 1507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:44599 every 8 connection(s)
I20260925 00:48:14.765733 1379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:14.772537 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 1379
I20260925 00:48:14.772660 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:14.772985 997 external_mini_cluster.cc:1428] Setting key 999cc1634f414fcaceb46c1739b4de5a
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:14.779008 1087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:14.767613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:42485 (local address 127.0.249.126:46163)
0925 00:48:14.767750 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:14.767753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:14.768424 (+ 671us) server_negotiation.cc:407] Connection header received
0925 00:48:14.769773 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:14.769776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:14.769820 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:14.769893 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:14.771056 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.771571 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.772890 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.773222 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.773422 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:14.776114 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:14.776117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:14.776119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:14.776142 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:14.777294 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.777786 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.777789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.777790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.777865 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.778152 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.778154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.778155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.778316 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:14.778390 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:14.778802 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 00:48:14.778895 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:48:14.779733 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:14.767867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46163 (local address 127.0.249.67:42485)
0925 00:48:14.768274 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:14.768316 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:14.769583 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:14.770085 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:14.770091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:14.770465 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:14.770927 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:14.770941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.771683 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.771685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:14.772767 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:14.772777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.773475 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.773481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:14.773699 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:14.774339 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:14.774354 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:14.775977 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:14.777423 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:14.777430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:14.777443 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:14.777699 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:14.777942 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:14.777944 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:14.777946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:14.778067 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:14.778395 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:14.778399 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:14.778606 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 00:48:14.778741 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 00:48:14.778959 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260925 00:48:14.780784 1508 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46163
I20260925 00:48:14.780990 1508 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:14.781435 1508 heartbeater.cc:511] Master 127.0.249.126:46163 requested a full tablet report, sending...
I20260925 00:48:14.782352 1049 ts_manager.cc:194] Registered new tserver with Master: 69f38db080bf45d2809db2ab93c88471 (127.0.249.67:44599)
I20260925 00:48:14.782851 1049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:42485
I20260925 00:48:14.787221 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:14.812217 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:14.802862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.0.249.65:33161)
0925 00:48:14.803186 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:14.803195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:14.803222 (+ 27us) server_negotiation.cc:407] Connection header received
0925 00:48:14.803270 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:14.803274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:14.803452 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:14.803567 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:14.804302 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.805080 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.806006 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.806368 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.806452 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:14.808793 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:14.808802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:14.808814 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:14.808836 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:14.810370 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.810806 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.810812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.810813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.810894 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.811207 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.811210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.811212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.811474 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:14.811683 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:14.811890 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 00:48:14.812046 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":115,"threads_started":1}
W20260925 00:48:14.813190 1154 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47100
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:14.838622 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:14.830176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.0.249.65:33161)
0925 00:48:14.830338 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:14.830344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:14.830493 (+ 149us) server_negotiation.cc:407] Connection header received
0925 00:48:14.830672 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:14.830674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:14.830718 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:14.830804 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:14.831602 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.832139 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.832977 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:14.833206 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:14.833316 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:14.835620 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:14.835629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:14.835631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:14.835661 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:14.836981 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.837376 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.837381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.837383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.837495 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:14.837769 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:14.837774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:14.837776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:14.837938 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:14.838070 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:14.838313 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 00:48:14.838460 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:48:14.839674 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 1111
I20260925 00:48:14.845795 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 1245
I20260925 00:48:14.852509 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 1379
I20260925 00:48:14.857626 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 1017
2026-09-25T00:48:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2167 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1522](info): set up 2 sockets
Sep 25 00:48:14 dist-test-slave-05m0 krb5kdc[1522](info): commencing operation
krb5kdc: starting...
W20260925 00:48:15.905661 997 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:15 dist-test-slave-05m0 krb5kdc[1522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:16.024669 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46243
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:16.131290 1535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:16.131543 1535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:16.131633 1535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:16.135578 1535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:16.135633 1535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:16.135653 1535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:16.135668 1535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:16.135684 1535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:16.140767 1535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:16.141778 1535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:16.142792 1535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:16.151265 1540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:16.151226 1541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:16.151242 1543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:16.151302 1535 server_base.cc:1034] running on GCE node
I20260925 00:48:16.151876 1535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:16.152908 1535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:16.154073 1535 hybrid_clock.cc:648] HybridClock initialized: now 1790297296154055 us; error 30 us; skew 500 ppm
Sep 25 00:48:16 dist-test-slave-05m0 krb5kdc[1522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:16.156781 1535 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:16.157729 1535 webserver.cc:460] Webserver started at http://127.0.249.126:36637/ using document root <none> and password file <none>
I20260925 00:48:16.158274 1535 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:16.158363 1535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:16.158589 1535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:16.160550 1535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "f154087a1c3546fdb2e66951b66328ca"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "31934092d17df59aa53c9b05a087d6c0"
server_key_iv: "efeb9558d34a542ee690be9a17b2ca52"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.161096 1535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "f154087a1c3546fdb2e66951b66328ca"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "31934092d17df59aa53c9b05a087d6c0"
server_key_iv: "efeb9558d34a542ee690be9a17b2ca52"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.164702 1535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 00:48:16.167028 1550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.168190 1535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:16.168330 1535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "f154087a1c3546fdb2e66951b66328ca"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "31934092d17df59aa53c9b05a087d6c0"
server_key_iv: "efeb9558d34a542ee690be9a17b2ca52"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.168437 1535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:16.182997 1535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:16.186228 1535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:16.186424 1535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:16.193384 1535 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:46243
I20260925 00:48:16.193428 1602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:46243 every 8 connection(s)
I20260925 00:48:16.194317 1535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:16.196867 1603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:16.200161 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 1535
I20260925 00:48:16.200281 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:16.200574 997 external_mini_cluster.cc:1428] Setting key 1bb96ab8fb57dfb08f16b12f8aadfcea
I20260925 00:48:16.202957 1603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca: Bootstrap starting.
I20260925 00:48:16.205085 1603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:16.205865 1603 log.cc:824] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:16.207998 1603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca: No bootstrap required, opened a new log
Sep 25 00:48:16 dist-test-slave-05m0 krb5kdc[1522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:16.211639 1603 raft_consensus.cc:348] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f154087a1c3546fdb2e66951b66328ca" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46243 } }
I20260925 00:48:16.211896 1603 raft_consensus.cc:374] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:16.211984 1603 raft_consensus.cc:729] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f154087a1c3546fdb2e66951b66328ca, State: Initialized, Role: FOLLOWER
I20260925 00:48:16.212407 1603 consensus_queue.cc:260] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f154087a1c3546fdb2e66951b66328ca" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46243 } }
I20260925 00:48:16.212532 1603 raft_consensus.cc:388] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:16.212620 1603 raft_consensus.cc:482] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:16.212719 1603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:16.213624 1603 raft_consensus.cc:504] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f154087a1c3546fdb2e66951b66328ca" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46243 } }
I20260925 00:48:16.213967 1603 leader_election.cc:302] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f154087a1c3546fdb2e66951b66328ca; no voters:
I20260925 00:48:16.214293 1603 leader_election.cc:288] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:16.214361 1608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:16.214576 1608 raft_consensus.cc:686] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [term 1 LEADER]: Becoming Leader. State: Replica: f154087a1c3546fdb2e66951b66328ca, State: Running, Role: LEADER
I20260925 00:48:16.214933 1608 consensus_queue.cc:237] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f154087a1c3546fdb2e66951b66328ca" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46243 } }
I20260925 00:48:16.215595 1603 sys_catalog.cc:564] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:16.215734 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.201985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.0.249.126:46243)
0925 00:48:16.202389 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:16.202403 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:16.202438 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:48:16.203089 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:16.203117 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:16.203427 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:16.203769 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:16.204549 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.205638 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.206665 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.207327 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.207517 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.210167 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:16.210172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:16.210184 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:16.210209 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:16.212624 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.213107 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.213113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.213119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.213234 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.213582 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.213585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.213587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.214027 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:16.214256 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:16.214594 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 00:48:16.214893 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260925 00:48:16.216696 1609 sys_catalog.cc:455] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f154087a1c3546fdb2e66951b66328ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f154087a1c3546fdb2e66951b66328ca" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46243 } } }
I20260925 00:48:16.216823 1609 sys_catalog.cc:458] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:16.217172 1610 sys_catalog.cc:455] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f154087a1c3546fdb2e66951b66328ca. Latest consensus state: current_term: 1 leader_uuid: "f154087a1c3546fdb2e66951b66328ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f154087a1c3546fdb2e66951b66328ca" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46243 } } }
I20260925 00:48:16.217281 1610 sys_catalog.cc:458] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:16.218339 1617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:16.220705 1617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:16.226266 1617 catalog_manager.cc:1347] Generated new cluster ID: f93eb4c98298419eb6f9b52777d9a465
I20260925 00:48:16.226330 1617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:16.237458 1617 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:16.238678 1617 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:16.249147 1617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f154087a1c3546fdb2e66951b66328ca: Generated new TSK 0
I20260925 00:48:16.250023 1617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:48:16.284282 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46243
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:16.394963 1629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:16.395205 1629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:16.395249 1629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:16.399289 1629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:16.399369 1629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:16.399461 1629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:16.404489 1629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46243
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:16.405545 1629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:16.406512 1629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:16.415540 1635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:16.415597 1634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:16.415676 1629 server_base.cc:1034] running on GCE node
W20260925 00:48:16.415540 1637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:16.416152 1629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:16.416795 1629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:16.417971 1629 hybrid_clock.cc:648] HybridClock initialized: now 1790297296417955 us; error 27 us; skew 500 ppm
Sep 25 00:48:16 dist-test-slave-05m0 krb5kdc[1522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:16.420531 1629 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:16.421368 1629 webserver.cc:460] Webserver started at http://127.0.249.65:41481/ using document root <none> and password file <none>
I20260925 00:48:16.421937 1629 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:16.422022 1629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:16.422264 1629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:16.424207 1629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "d319d7ae6bb74521b9bbab28519f63c3"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "61ccd4f390617b973e55c3bd3ea8c41d"
server_key_iv: "6aa0abb4a3127205e1f38bd759a8ede2"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.424767 1629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "d319d7ae6bb74521b9bbab28519f63c3"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "61ccd4f390617b973e55c3bd3ea8c41d"
server_key_iv: "6aa0abb4a3127205e1f38bd759a8ede2"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.428264 1629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:48:16.430536 1644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.431694 1629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:16.431821 1629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "d319d7ae6bb74521b9bbab28519f63c3"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "61ccd4f390617b973e55c3bd3ea8c41d"
server_key_iv: "6aa0abb4a3127205e1f38bd759a8ede2"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.431928 1629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:16.444109 1629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:16.446812 1629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:16.447001 1629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:16.447536 1629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:16.448311 1629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:16.448377 1629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.448441 1629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:16.448486 1629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.457655 1629 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:34273
I20260925 00:48:16.457690 1757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:34273 every 8 connection(s)
I20260925 00:48:16.458516 1629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:16.460314 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 1629
I20260925 00:48:16.460419 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:16.460673 997 external_mini_cluster.cc:1428] Setting key 4be6fed9ba4b51bd147fe9971482ee37
Sep 25 00:48:16 dist-test-slave-05m0 krb5kdc[1522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
I20260925 00:48:16.474014 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.460260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:42685 (local address 127.0.249.126:46243)
0925 00:48:16.460410 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:16.460414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:16.461270 (+ 856us) server_negotiation.cc:407] Connection header received
0925 00:48:16.462658 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:16.462661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:16.462712 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:16.462796 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:16.464156 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.464686 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.466489 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.466719 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.466799 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.469729 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:16.469733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:16.469735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:16.469759 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:16.471288 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.472018 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.472022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.472023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.472118 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.472611 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.472614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.472615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.472798 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:16.472874 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:16.473617 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 00:48:16.473764 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 00:48:16.474823 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.460622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46243 (local address 127.0.249.65:42685)
0925 00:48:16.461113 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:16.461165 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:16.462453 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:16.462976 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:16.462983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:16.463318 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:16.463996 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.464018 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.464797 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.464803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.466364 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.466379 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.466872 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.466880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.467173 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.467810 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:16.467837 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:16.469557 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:16.471503 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:16.471513 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:16.471549 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:16.471876 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:16.472215 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:16.472217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:16.472219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:16.472514 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:16.472918 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:16.472923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:16.473281 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 00:48:16.473536 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 00:48:16.473833 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":138,"threads_started":1}
I20260925 00:48:16.476323 1758 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46243
I20260925 00:48:16.476585 1758 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:16.477221 1758 heartbeater.cc:511] Master 127.0.249.126:46243 requested a full tablet report, sending...
I20260925 00:48:16.478866 1567 ts_manager.cc:194] Registered new tserver with Master: d319d7ae6bb74521b9bbab28519f63c3 (127.0.249.65:34273)
I20260925 00:48:16.480041 1567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:42685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:16.489082 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46243
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:16.597103 1763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:16.597409 1763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:16.597537 1763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:16.601485 1763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:16.601575 1763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:16.601670 1763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:16.606755 1763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46243
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:16.607765 1763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:16.608732 1763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:16.617916 1768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:16.617913 1771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:16.618052 1769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:16.618491 1763 server_base.cc:1034] running on GCE node
I20260925 00:48:16.618906 1763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:16.619578 1763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:16.620771 1763 hybrid_clock.cc:648] HybridClock initialized: now 1790297296620753 us; error 34 us; skew 500 ppm
Sep 25 00:48:16 dist-test-slave-05m0 krb5kdc[1522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:16.623535 1763 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:16.624461 1763 webserver.cc:460] Webserver started at http://127.0.249.66:44439/ using document root <none> and password file <none>
I20260925 00:48:16.625027 1763 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:16.625124 1763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:16.625372 1763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:16.627444 1763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "683f66e461ed476f85cb23f2c28914c7"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "6a0f09fa5c7e3718e952de6e974ecbf5"
server_key_iv: "bfc44633eaa6fc0cb5eaa21052db1530"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.628032 1763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "683f66e461ed476f85cb23f2c28914c7"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "6a0f09fa5c7e3718e952de6e974ecbf5"
server_key_iv: "bfc44633eaa6fc0cb5eaa21052db1530"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.631861 1763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 00:48:16.634258 1778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.635406 1763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:16.635552 1763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "683f66e461ed476f85cb23f2c28914c7"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "6a0f09fa5c7e3718e952de6e974ecbf5"
server_key_iv: "bfc44633eaa6fc0cb5eaa21052db1530"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.635671 1763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:16.649470 1763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:16.652130 1763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:16.652329 1763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:16.652902 1763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:16.653795 1763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:16.653863 1763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.653932 1763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:16.653966 1763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.662492 1763 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:43559
I20260925 00:48:16.662514 1891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:43559 every 8 connection(s)
I20260925 00:48:16.663353 1763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:16.664762 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 1763
I20260925 00:48:16.664883 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:16.665206 997 external_mini_cluster.cc:1428] Setting key 402523d076541d32c378f444bd64e1df
Sep 25 00:48:16 dist-test-slave-05m0 krb5kdc[1522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
I20260925 00:48:16.677518 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.665244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:51685 (local address 127.0.249.126:46243)
0925 00:48:16.665409 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:16.665436 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:16.666386 (+ 950us) server_negotiation.cc:407] Connection header received
0925 00:48:16.667885 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:16.667888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:16.667949 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:16.668026 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:16.669136 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.669746 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.670945 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.671171 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.671281 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.674021 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:16.674023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:16.674025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:16.674049 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:16.675362 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.675822 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.675824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.675825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.675893 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.676298 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.676300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.676301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.676472 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:16.676548 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:16.677207 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 00:48:16.677324 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:48:16.678246 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.665763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46243 (local address 127.0.249.66:51685)
0925 00:48:16.666239 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:16.666288 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:16.667683 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:16.668197 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:16.668203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:16.668545 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:16.669001 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.669013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.669870 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.669876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.670812 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.670821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.671314 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.671319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.671490 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.672096 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:16.672111 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:16.673868 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:16.675464 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:16.675469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:16.675479 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:16.675737 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:16.676017 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:16.676023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:16.676028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:16.676204 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:16.676586 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:16.676594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:16.676918 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 00:48:16.677146 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 00:48:16.677470 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
I20260925 00:48:16.679512 1892 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46243
I20260925 00:48:16.679746 1892 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:16.680229 1892 heartbeater.cc:511] Master 127.0.249.126:46243 requested a full tablet report, sending...
I20260925 00:48:16.681331 1567 ts_manager.cc:194] Registered new tserver with Master: 683f66e461ed476f85cb23f2c28914c7 (127.0.249.66:43559)
I20260925 00:48:16.682267 1567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:51685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:16.691972 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46243
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:16.800012 1897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:16.800259 1897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:16.800302 1897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:16.804266 1897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:16.804319 1897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:16.804402 1897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:16.809466 1897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46243
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:16.810446 1897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:16.811404 1897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:16.820842 1903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:16.820856 1905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:16.820956 1902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:16.821125 1897 server_base.cc:1034] running on GCE node
I20260925 00:48:16.821590 1897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:16.822218 1897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:16.823407 1897 hybrid_clock.cc:648] HybridClock initialized: now 1790297296823394 us; error 31 us; skew 500 ppm
Sep 25 00:48:16 dist-test-slave-05m0 krb5kdc[1522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:16.826020 1897 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:16.826900 1897 webserver.cc:460] Webserver started at http://127.0.249.67:34425/ using document root <none> and password file <none>
I20260925 00:48:16.827440 1897 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:16.827528 1897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:16.827761 1897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:16.829730 1897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "dcac0ae1cb4b4f6bb523a72c3675963f"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "6cf1ccc0c50814a83c2887e631d26e54"
server_key_iv: "0413d4642a500c4be19e2f471ba8ac22"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.830272 1897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "dcac0ae1cb4b4f6bb523a72c3675963f"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "6cf1ccc0c50814a83c2887e631d26e54"
server_key_iv: "0413d4642a500c4be19e2f471ba8ac22"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.833735 1897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:16.836011 1912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.837221 1897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:16.837340 1897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "dcac0ae1cb4b4f6bb523a72c3675963f"
format_stamp: "Formatted at 2026-09-25 00:48:16 on dist-test-slave-05m0"
server_key: "6cf1ccc0c50814a83c2887e631d26e54"
server_key_iv: "0413d4642a500c4be19e2f471ba8ac22"
server_key_version: "encryptionkey@0"
I20260925 00:48:16.837483 1897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:16.853127 1897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:16.855841 1897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:16.856030 1897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:16.856588 1897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:16.857427 1897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:16.857491 1897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.857563 1897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:16.857610 1897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:16.868315 1897 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:34755
I20260925 00:48:16.868397 2025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:34755 every 8 connection(s)
I20260925 00:48:16.869392 1897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:16.877912 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 1897
I20260925 00:48:16.878046 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:16.878399 997 external_mini_cluster.cc:1428] Setting key 46dbe6eaef223e821602adcc1bf8447e
Sep 25 00:48:16 dist-test-slave-05m0 krb5kdc[1522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:16.885114 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.871778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:55803 (local address 127.0.249.126:46243)
0925 00:48:16.871949 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:16.871972 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:16.873315 (+ 1343us) server_negotiation.cc:407] Connection header received
0925 00:48:16.874318 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:16.874321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:16.874368 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:16.874443 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:16.875605 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.876113 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.877331 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.877617 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.877695 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.881022 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:16.881025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:16.881026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:16.881050 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:16.883008 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.883600 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.883603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.883605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.883690 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.884020 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.884024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.884026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.884214 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:16.884319 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:16.884823 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 00:48:16.884941 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:48:16.885958 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.872146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46243 (local address 127.0.249.67:55803)
0925 00:48:16.873166 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:16.873211 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:16.874093 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:16.874579 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:16.874586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:16.874974 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:16.875478 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.875492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.876230 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.876233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.877227 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.877236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.877930 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.877940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.878169 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.878825 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:16.878842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:16.880863 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:16.883124 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:16.883130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:16.883145 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:16.883449 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:16.883780 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:16.883785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:16.883787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:16.883927 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:16.884316 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:16.884322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:16.884592 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 00:48:16.884781 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 00:48:16.885092 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":589,"threads_started":1}
I20260925 00:48:16.887109 2026 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46243
I20260925 00:48:16.887331 2026 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:16.887779 2026 heartbeater.cc:511] Master 127.0.249.126:46243 requested a full tablet report, sending...
I20260925 00:48:16.888823 1567 ts_manager.cc:194] Registered new tserver with Master: dcac0ae1cb4b4f6bb523a72c3675963f (127.0.249.67:34755)
I20260925 00:48:16.889344 1567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:55803
I20260925 00:48:16.892321 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:48:16.902278 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.895612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.0.249.126:46243)
0925 00:48:16.895734 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:16.895737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:16.895836 (+ 99us) server_negotiation.cc:407] Connection header received
0925 00:48:16.895930 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:16.895933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:16.895975 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:16.896047 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:16.896681 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.897207 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.898055 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.898257 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.898337 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.899565 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:16.899568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:16.899570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:16.899598 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:16.900869 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.901241 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.901244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.901245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.901315 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:16.901594 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:16.901596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:16.901597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:16.901793 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:16.901895 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:16.902079 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 00:48:16.902166 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:16.906804 1567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:16.909816 1567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:16.923836 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.917753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:34273 (local address 127.0.0.1:45852)
0925 00:48:16.918520 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:16.918573 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:16.918696 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:16.919834 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:16.919837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:16.919862 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:16.919982 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.919990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.921031 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.921035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.922272 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.922281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.923280 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.923284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.923391 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.923460 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 00:48:16.923565 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 00:48:16.923687 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":98,"threads_started":1}
I20260925 00:48:16.924024 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.918096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.0.249.67:34755)
0925 00:48:16.919168 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:16.919184 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:16.919207 (+ 23us) server_negotiation.cc:407] Connection header received
0925 00:48:16.919280 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:16.919285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:16.919492 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:16.919655 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:16.919955 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.921158 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.922763 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.923391 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.923503 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.923651 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:16.923750 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 00:48:16.923837 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":115,"threads_started":1}
I20260925 00:48:16.924453 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.917865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.0.249.65:34273)
0925 00:48:16.918676 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:16.918687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:16.918712 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:48:16.918899 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:16.918905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:16.919494 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:16.919655 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:16.920107 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.920879 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.922401 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.923138 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.923246 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.923388 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:16.924251 (+ 863us) server_negotiation.cc:299] Negotiation successful
0925 00:48:16.924335 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":142,"threads_started":1}
I20260925 00:48:16.924612 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.919048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.0.249.66:43559)
0925 00:48:16.919398 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:16.919409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:16.919429 (+ 20us) server_negotiation.cc:407] Connection header received
0925 00:48:16.919497 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:16.919501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:16.919688 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:16.919836 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:16.920240 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.921326 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.922786 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.923575 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.923672 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.923812 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:16.924320 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 00:48:16.924432 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":140,"threads_started":1}
I20260925 00:48:16.923985 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.917919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:34755 (local address 127.0.0.1:57550)
0925 00:48:16.918846 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:16.918858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:16.918931 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:16.919654 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:16.919660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:16.919682 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:16.919829 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.919836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.921312 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.921315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.922601 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.922612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.923518 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.923522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.923619 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.923655 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 00:48:16.923725 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:48:16.923794 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260925 00:48:16.924248 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:16.918641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:43559 (local address 127.0.0.1:55654)
0925 00:48:16.919218 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:16.919234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:16.919321 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:16.920002 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:16.920004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:16.920018 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:16.920108 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.920114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.921534 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.921538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.922637 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:16.922647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:16.923870 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:16.923873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:16.923968 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:16.923989 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:48:16.924060 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 00:48:16.924141 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260925 00:48:16.927330 1692 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4edd05decc4605a6343c29a9aec94a (DEFAULT_TABLE table=test-table [id=25b43e9257b24d0f8a03f02c9bc086fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:16.927507 1960 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4edd05decc4605a6343c29a9aec94a (DEFAULT_TABLE table=test-table [id=25b43e9257b24d0f8a03f02c9bc086fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:16.927848 1826 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4edd05decc4605a6343c29a9aec94a (DEFAULT_TABLE table=test-table [id=25b43e9257b24d0f8a03f02c9bc086fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:16.928309 1692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4edd05decc4605a6343c29a9aec94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:16.928499 1960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4edd05decc4605a6343c29a9aec94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:16.928763 1826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4edd05decc4605a6343c29a9aec94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:16.933972 2043 tablet_bootstrap.cc:492] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3: Bootstrap starting.
I20260925 00:48:16.935362 2044 tablet_bootstrap.cc:492] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f: Bootstrap starting.
I20260925 00:48:16.936120 2045 tablet_bootstrap.cc:492] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7: Bootstrap starting.
I20260925 00:48:16.936668 2043 tablet_bootstrap.cc:654] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:16.937569 2044 tablet_bootstrap.cc:654] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:16.938118 2043 log.cc:824] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:16.938272 2045 tablet_bootstrap.cc:654] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:16.938822 2044 log.cc:824] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:16.939105 2045 log.cc:824] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:16.940075 2043 tablet_bootstrap.cc:492] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3: No bootstrap required, opened a new log
I20260925 00:48:16.940259 2043 ts_tablet_manager.cc:1397] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:48:16.941052 2045 tablet_bootstrap.cc:492] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7: No bootstrap required, opened a new log
I20260925 00:48:16.941092 2044 tablet_bootstrap.cc:492] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f: No bootstrap required, opened a new log
I20260925 00:48:16.941227 2045 ts_tablet_manager.cc:1397] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:48:16.941268 2044 ts_tablet_manager.cc:1397] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:48:16.944036 2044 raft_consensus.cc:348] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } }
I20260925 00:48:16.944065 2043 raft_consensus.cc:348] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } }
I20260925 00:48:16.944278 2044 raft_consensus.cc:374] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:16.944295 2043 raft_consensus.cc:374] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:16.944339 2044 raft_consensus.cc:729] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcac0ae1cb4b4f6bb523a72c3675963f, State: Initialized, Role: FOLLOWER
I20260925 00:48:16.944348 2043 raft_consensus.cc:729] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d319d7ae6bb74521b9bbab28519f63c3, State: Initialized, Role: FOLLOWER
I20260925 00:48:16.944775 2044 consensus_queue.cc:260] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } }
I20260925 00:48:16.944802 2043 consensus_queue.cc:260] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } }
I20260925 00:48:16.944778 2045 raft_consensus.cc:348] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } }
I20260925 00:48:16.944993 2045 raft_consensus.cc:374] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:16.945068 2045 raft_consensus.cc:729] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683f66e461ed476f85cb23f2c28914c7, State: Initialized, Role: FOLLOWER
I20260925 00:48:16.945462 2045 consensus_queue.cc:260] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } }
I20260925 00:48:16.945777 2043 ts_tablet_manager.cc:1428] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:48:16.945804 2044 ts_tablet_manager.cc:1428] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 00:48:16.946147 1758 heartbeater.cc:503] Master 127.0.249.126:46243 was elected leader, sending a full tablet report...
I20260925 00:48:16.946170 2026 heartbeater.cc:503] Master 127.0.249.126:46243 was elected leader, sending a full tablet report...
I20260925 00:48:16.946292 2045 ts_tablet_manager.cc:1428] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:48:16.947122 1892 heartbeater.cc:503] Master 127.0.249.126:46243 was elected leader, sending a full tablet report...
W20260925 00:48:16.959568 1759 tablet.cc:2369] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:17.072853 2049 raft_consensus.cc:482] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:17.073076 2049 raft_consensus.cc:504] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } }
I20260925 00:48:17.074307 2049 leader_election.cc:288] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d319d7ae6bb74521b9bbab28519f63c3 (127.0.249.65:34273), 683f66e461ed476f85cb23f2c28914c7 (127.0.249.66:43559)
I20260925 00:48:17.077939 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:17.074629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:49459 (local address 127.0.249.65:34273)
0925 00:48:17.074782 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:17.074786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:17.074800 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:17.074948 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:17.074951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:17.075018 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:17.075099 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:17.075412 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.075938 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.076989 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.077366 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.077520 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:17.077574 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:17.077759 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 00:48:17.077828 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 00:48:17.077950 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:17.074500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:34273 (local address 127.0.249.67:49459)
0925 00:48:17.074697 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:17.074714 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:17.074813 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:17.075153 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:17.075156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:17.075175 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:17.075296 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:17.075302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.076045 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.076051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:17.076846 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:17.076855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.077508 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.077512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:17.077616 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:17.077638 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:48:17.077733 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 00:48:17.077805 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":2}
I20260925 00:48:17.078402 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:17.074988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:36025 (local address 127.0.249.66:43559)
0925 00:48:17.075128 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:17.075133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:17.075328 (+ 195us) server_negotiation.cc:407] Connection header received
0925 00:48:17.075452 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:17.075456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:17.075510 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:17.075597 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:17.075874 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.076381 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.077446 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.077852 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.077983 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:17.078028 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:17.078240 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 00:48:17.078304 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:48:17.078411 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:17.074847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:43559 (local address 127.0.249.67:36025)
0925 00:48:17.075237 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:17.075248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:17.075360 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:17.075608 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:17.075611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:17.075636 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:17.075766 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:17.075771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.076493 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.076498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:17.077308 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:17.077316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.077983 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.077987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:17.078077 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:17.078104 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:48:17.078212 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 00:48:17.078276 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":24,"thread_start_us":140,"threads_started":1}
I20260925 00:48:17.078585 1712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4edd05decc4605a6343c29a9aec94a" candidate_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d319d7ae6bb74521b9bbab28519f63c3" is_pre_election: true
I20260925 00:48:17.078895 1712 raft_consensus.cc:2454] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcac0ae1cb4b4f6bb523a72c3675963f in term 0.
I20260925 00:48:17.079375 1914 leader_election.cc:302] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d319d7ae6bb74521b9bbab28519f63c3, dcac0ae1cb4b4f6bb523a72c3675963f; no voters:
I20260925 00:48:17.079648 2049 raft_consensus.cc:2781] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:17.079748 2049 raft_consensus.cc:482] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:17.079668 1846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4edd05decc4605a6343c29a9aec94a" candidate_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683f66e461ed476f85cb23f2c28914c7" is_pre_election: true
I20260925 00:48:17.079818 2049 raft_consensus.cc:3037] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:17.080015 1846 raft_consensus.cc:2454] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcac0ae1cb4b4f6bb523a72c3675963f in term 0.
I20260925 00:48:17.081012 2049 raft_consensus.cc:504] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } }
I20260925 00:48:17.081379 2049 leader_election.cc:288] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [CANDIDATE]: Term 1 election: Requested vote from peers d319d7ae6bb74521b9bbab28519f63c3 (127.0.249.65:34273), 683f66e461ed476f85cb23f2c28914c7 (127.0.249.66:43559)
I20260925 00:48:17.081703 1712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4edd05decc4605a6343c29a9aec94a" candidate_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d319d7ae6bb74521b9bbab28519f63c3"
I20260925 00:48:17.081816 1846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4edd05decc4605a6343c29a9aec94a" candidate_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683f66e461ed476f85cb23f2c28914c7"
I20260925 00:48:17.081801 1712 raft_consensus.cc:3037] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:17.081946 1846 raft_consensus.cc:3037] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:17.082842 1712 raft_consensus.cc:2454] T 0e4edd05decc4605a6343c29a9aec94a P d319d7ae6bb74521b9bbab28519f63c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcac0ae1cb4b4f6bb523a72c3675963f in term 1.
I20260925 00:48:17.082988 1846 raft_consensus.cc:2454] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcac0ae1cb4b4f6bb523a72c3675963f in term 1.
I20260925 00:48:17.083210 1914 leader_election.cc:302] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d319d7ae6bb74521b9bbab28519f63c3, dcac0ae1cb4b4f6bb523a72c3675963f; no voters:
I20260925 00:48:17.083356 2049 raft_consensus.cc:2781] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:17.083573 2049 raft_consensus.cc:686] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [term 1 LEADER]: Becoming Leader. State: Replica: dcac0ae1cb4b4f6bb523a72c3675963f, State: Running, Role: LEADER
I20260925 00:48:17.083905 2049 consensus_queue.cc:237] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } }
I20260925 00:48:17.087074 1565 catalog_manager.cc:5495] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f reported cstate change: term changed from 0 to 1, leader changed from <none> to dcac0ae1cb4b4f6bb523a72c3675963f (127.0.249.67). New cstate: current_term: 1 leader_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d319d7ae6bb74521b9bbab28519f63c3" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcac0ae1cb4b4f6bb523a72c3675963f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "683f66e461ed476f85cb23f2c28914c7" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43559 } health_report { overall_health: UNKNOWN } } }
Sep 25 00:48:17 dist-test-slave-05m0 krb5kdc[1522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:17.120270 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:17.111006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.0.249.65:34273)
0925 00:48:17.111181 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:17.111186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:17.111358 (+ 172us) server_negotiation.cc:407] Connection header received
0925 00:48:17.111531 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:17.111539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:17.111593 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:17.111685 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:17.112412 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.112968 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.113857 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.114080 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.114188 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:17.116675 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:17.116688 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:17.116704 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:17.116728 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:17.118458 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:17.118902 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:17.118909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:17.118910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:17.118992 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:17.119287 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:17.119292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:17.119294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:17.119564 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:17.119790 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:17.119954 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 00:48:17.120079 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 00:48:17.120472 2027 tablet.cc:2369] T 0e4edd05decc4605a6343c29a9aec94a P dcac0ae1cb4b4f6bb523a72c3675963f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 00:48:17 dist-test-slave-05m0 krb5kdc[1522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:48:17 dist-test-slave-05m0 krb5kdc[1522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:17.147440 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:17.138360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.0.249.65:34273)
0925 00:48:17.138502 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:17.138506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:17.138538 (+ 32us) server_negotiation.cc:407] Connection header received
0925 00:48:17.138660 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:17.138663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:17.138706 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:17.138771 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:17.139544 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.140281 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.141359 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.141721 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.141813 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:17.144422 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:17.144425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:17.144427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:17.144448 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:17.145908 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:17.146290 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:17.146292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:17.146294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:17.146363 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:17.146588 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:17.146590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:17.146591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:17.146730 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:17.146841 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:17.147137 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 00:48:17.147228 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 00:48:17 dist-test-slave-05m0 krb5kdc[1522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260925 00:48:17.164572 1893 tablet.cc:2369] T 0e4edd05decc4605a6343c29a9aec94a P 683f66e461ed476f85cb23f2c28914c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 00:48:17 dist-test-slave-05m0 krb5kdc[1522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:17.174326 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:17.165755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.0.249.65:34273)
0925 00:48:17.165931 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:17.165935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:17.166085 (+ 150us) server_negotiation.cc:407] Connection header received
0925 00:48:17.166182 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:17.166184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:17.166226 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:17.166304 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:17.166955 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.167457 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.168261 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:17.168490 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:17.168573 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:17.171030 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:17.171033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:17.171036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:17.171059 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:17.172436 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:17.172972 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:17.172977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:17.172979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:17.173058 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:17.173327 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:17.173331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:17.173333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:17.173622 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:17.173742 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:17.174003 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 00:48:17.174115 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 00:48:17.175603 1672 tablet_service.cc:2908] Rejecting scan request for tablet 0e4edd05decc4605a6343c29a9aec94a: Uninitialized: clean time has not yet been initialized
I20260925 00:48:17.177654 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 1629
I20260925 00:48:17.183344 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 1763
I20260925 00:48:17.188284 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 1897
I20260925 00:48:17.193635 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 1535
2026-09-25T00:48:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2336 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:17 dist-test-slave-05m0 krb5kdc[2067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:17 dist-test-slave-05m0 krb5kdc[2067](info): set up 2 sockets
Sep 25 00:48:17 dist-test-slave-05m0 krb5kdc[2067](info): commencing operation
krb5kdc: starting...
W20260925 00:48:18.295225 997 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 00:48:18.534627 997 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.239s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 00:48:18.763232 997 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.228s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 00:48:18.906320 997 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.143s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/test-user.keytab.
W20260925 00:48:19.806519 997 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.900s user 0.001s sys 0.001s
Sep 25 00:48:19 dist-test-slave-05m0 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
W20260925 00:48:21.043992 997 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.249.126: real 1.215s user 0.000s sys 0.003s
I20260925 00:48:21.044673 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:41735
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:40109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:41735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:21.156668 2080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:21.156983 2080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:21.157084 2080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:21.161129 2080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:21.161217 2080 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 00:48:21.161253 2080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:21.161322 2080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:21.161402 2080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:21.161463 2080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:21.166795 2080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:40109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:41735
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:41735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:21.167850 2080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:21.168866 2080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:21.177567 2085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:21.177795 2080 server_base.cc:1034] running on GCE node
W20260925 00:48:21.177567 2086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:21.177644 2088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:21.178465 2080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:21.179576 2080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:21.180784 2080 hybrid_clock.cc:648] HybridClock initialized: now 1790297301180751 us; error 49 us; skew 500 ppm
Sep 25 00:48:21 dist-test-slave-05m0 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:21.188063 2080 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:21.188947 2080 webserver.cc:460] Webserver started at http://127.0.249.126:43121/ using document root <none> and password file <none>
I20260925 00:48:21.189558 2080 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:21.189623 2080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:21.189802 2080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:21.191845 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "493dd2460b6140acb4e47a8936123976"
format_stamp: "Formatted at 2026-09-25 00:48:21 on dist-test-slave-05m0"
server_key: "52940764f6e6e0f39aec566dd2bf44a4"
server_key_iv: "b1c6f42c813a401b73424623c275a686"
server_key_version: "encryptionkey@0"
I20260925 00:48:21.192631 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "493dd2460b6140acb4e47a8936123976"
format_stamp: "Formatted at 2026-09-25 00:48:21 on dist-test-slave-05m0"
server_key: "52940764f6e6e0f39aec566dd2bf44a4"
server_key_iv: "b1c6f42c813a401b73424623c275a686"
server_key_version: "encryptionkey@0"
I20260925 00:48:21.196578 2080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:48:21.199105 2095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:21.200944 2080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:48:21.201087 2080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "493dd2460b6140acb4e47a8936123976"
format_stamp: "Formatted at 2026-09-25 00:48:21 on dist-test-slave-05m0"
server_key: "52940764f6e6e0f39aec566dd2bf44a4"
server_key_iv: "b1c6f42c813a401b73424623c275a686"
server_key_version: "encryptionkey@0"
I20260925 00:48:21.201192 2080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:21.217175 2080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:21.220736 2080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:21.220954 2080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:21.229106 2080 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:41735
I20260925 00:48:21.229125 2157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:41735 every 8 connection(s)
I20260925 00:48:21.230114 2080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:21.230948 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 2080
I20260925 00:48:21.231077 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:21.231317 997 external_mini_cluster.cc:1428] Setting key 78be2d4edccccad9b0c67c47f8956e8e
I20260925 00:48:21.233445 2158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:21.238997 2158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976: Bootstrap starting.
Sep 25 00:48:21 dist-test-slave-05m0 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:21.241375 2158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:21.242219 2158 log.cc:824] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:21.244309 2158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976: No bootstrap required, opened a new log
I20260925 00:48:21.247466 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:21.232701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.0.249.126:41735)
0925 00:48:21.233158 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:21.233174 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:21.233209 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:48:21.234052 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:21.234072 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:21.234424 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:21.234816 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:21.235650 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.236769 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.237707 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.238223 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.238438 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:21.241850 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:21.241856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:21.241871 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:21.241904 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:21.244350 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:21.244970 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:21.244975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:21.244980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:21.245100 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:21.245486 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:21.245490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:21.245492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:21.245919 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:21.246088 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:21.246402 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 00:48:21.246699 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260925 00:48:21.247975 2158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493dd2460b6140acb4e47a8936123976" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41735 } }
I20260925 00:48:21.248229 2158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:21.248323 2158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493dd2460b6140acb4e47a8936123976, State: Initialized, Role: FOLLOWER
I20260925 00:48:21.248826 2158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493dd2460b6140acb4e47a8936123976" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41735 } }
I20260925 00:48:21.248991 2158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:21.249056 2158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:21.249150 2158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:21.312071 2158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493dd2460b6140acb4e47a8936123976" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41735 } }
I20260925 00:48:21.312577 2158 leader_election.cc:302] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 493dd2460b6140acb4e47a8936123976; no voters:
I20260925 00:48:21.312969 2158 leader_election.cc:288] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:21.313117 2163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:21.313365 2163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [term 1 LEADER]: Becoming Leader. State: Replica: 493dd2460b6140acb4e47a8936123976, State: Running, Role: LEADER
I20260925 00:48:21.313864 2163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493dd2460b6140acb4e47a8936123976" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41735 } }
I20260925 00:48:21.314477 2158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:21.315637 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "493dd2460b6140acb4e47a8936123976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493dd2460b6140acb4e47a8936123976" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41735 } } }
I20260925 00:48:21.315774 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:21.316231 2165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493dd2460b6140acb4e47a8936123976. Latest consensus state: current_term: 1 leader_uuid: "493dd2460b6140acb4e47a8936123976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493dd2460b6140acb4e47a8936123976" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41735 } } }
I20260925 00:48:21.316293 2172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:21.316398 2165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:21.319062 2172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:21.325615 2172 catalog_manager.cc:1347] Generated new cluster ID: 9573feed91974eb0a0fa364951e94c5c
I20260925 00:48:21.325687 2172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:21.344437 2172 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:21.345498 2172 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:21.349889 2172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976: Generated new TSK 0
I20260925 00:48:21.351044 2172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
W20260925 00:48:21.746107 997 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.249.65: real 0.379s user 0.000s sys 0.004s
I20260925 00:48:21.746603 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:41735
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:40109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:21.853878 2184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:21.854153 2184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:21.854230 2184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:21.858274 2184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:21.858335 2184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:21.858402 2184 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:48:21.858486 2184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:21.863812 2184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:40109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.249.126:41735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:21.864854 2184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:21.865949 2184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:21.875850 2192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:21.875998 2189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:21.876165 2184 server_base.cc:1034] running on GCE node
W20260925 00:48:21.875854 2190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:21.876730 2184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:21.877547 2184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:21.878746 2184 hybrid_clock.cc:648] HybridClock initialized: now 1790297301878720 us; error 48 us; skew 500 ppm
Sep 25 00:48:21 dist-test-slave-05m0 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:21.881675 2184 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:21.882499 2184 webserver.cc:460] Webserver started at http://127.0.249.65:34881/ using document root <none> and password file <none>
I20260925 00:48:21.882988 2184 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:21.883052 2184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:21.883209 2184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:21.885164 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "aa23292df91b4f0092147676cf979b3f"
format_stamp: "Formatted at 2026-09-25 00:48:21 on dist-test-slave-05m0"
server_key: "29f7382bcd061a554f95f1b60acd2603"
server_key_iv: "182ac284b9d734e3f667414df4feb497"
server_key_version: "encryptionkey@0"
I20260925 00:48:21.885741 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "aa23292df91b4f0092147676cf979b3f"
format_stamp: "Formatted at 2026-09-25 00:48:21 on dist-test-slave-05m0"
server_key: "29f7382bcd061a554f95f1b60acd2603"
server_key_iv: "182ac284b9d734e3f667414df4feb497"
server_key_version: "encryptionkey@0"
I20260925 00:48:21.889179 2184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:21.891490 2199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:21.892686 2184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:48:21.892808 2184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "aa23292df91b4f0092147676cf979b3f"
format_stamp: "Formatted at 2026-09-25 00:48:21 on dist-test-slave-05m0"
server_key: "29f7382bcd061a554f95f1b60acd2603"
server_key_iv: "182ac284b9d734e3f667414df4feb497"
server_key_version: "encryptionkey@0"
I20260925 00:48:21.892902 2184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:21.908337 2184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:21.911118 2184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:21.911303 2184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:21.912714 2184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:21.912784 2184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:21.912851 2184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:21.912902 2184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:48:21 dist-test-slave-05m0 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:21.923386 2184 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:41613
I20260925 00:48:21.923475 2315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:41613 every 8 connection(s)
I20260925 00:48:21.924376 2184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:21.926713 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:21.913462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:43827 (local address 127.0.249.126:41735)
0925 00:48:21.913699 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:21.913703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:21.914635 (+ 932us) server_negotiation.cc:407] Connection header received
0925 00:48:21.915652 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:21.915660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:21.915733 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:21.915853 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:21.917579 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.918113 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.919944 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.920341 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.920483 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:21.923049 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:21.923052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:21.923055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:21.923085 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:21.924803 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:21.925373 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:21.925376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:21.925377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:21.925509 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:21.925809 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:21.925811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:21.925812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:21.925986 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:21.926063 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:21.926467 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 00:48:21.926590 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260925 00:48:21.927462 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:21.913984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41735 (local address 127.0.249.65:43827)
0925 00:48:21.914496 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:21.914543 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:21.915392 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:21.916182 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:21.916193 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:21.916693 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:21.917341 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:21.917366 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.918220 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.918223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:21.919805 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:21.919822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.920521 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.920525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:21.920709 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:21.921322 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:21.921341 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:21.922854 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:21.924939 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:21.924944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:21.924958 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:21.925251 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:21.925594 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:21.925596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:21.925598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:21.925721 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:21.926065 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:21.926070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:21.926270 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 00:48:21.926407 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 00:48:21.926650 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260925 00:48:21.928615 2316 heartbeater.cc:347] Connected to a master server at 127.0.249.126:41735
I20260925 00:48:21.928828 2316 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:21.929280 2316 heartbeater.cc:511] Master 127.0.249.126:41735 requested a full tablet report, sending...
I20260925 00:48:21.930806 2112 ts_manager.cc:194] Registered new tserver with Master: aa23292df91b4f0092147676cf979b3f (127.0.249.65:41613)
I20260925 00:48:21.931898 2112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:43827
I20260925 00:48:21.932813 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 2184
I20260925 00:48:21.932926 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:21.933243 997 external_mini_cluster.cc:1428] Setting key 03dd1201e72c307f65bfdb9c20e70c29
I20260925 00:48:21.939787 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:21.930841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:48559 (local address 127.0.249.126:41735)
0925 00:48:21.932478 (+ 1637us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:21.932484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:21.932496 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:21.932526 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:21.932528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:21.932599 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:21.932683 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:21.933217 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.933962 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.934955 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.935352 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.935476 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:21.936572 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:21.936577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:21.936581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:21.936610 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:21.938118 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:21.938627 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:21.938632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:21.938637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:21.938731 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:21.939044 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:21.939046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:21.939047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:21.939202 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:21.939276 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:21.939584 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 00:48:21.939679 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1559}
I20260925 00:48:21.939978 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:21.930773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41735 (local address 127.0.249.65:48559)
0925 00:48:21.931065 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:21.931078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:21.931145 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:21.932722 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:21.932726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:21.932965 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:21.933091 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:21.933100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.934098 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.934101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:21.934840 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:21.934850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:21.935469 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:21.935472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:21.935537 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:21.936081 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:21.936101 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:21.936436 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:21.938232 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:21.938238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:21.938243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:21.938514 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:21.938832 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:21.938838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:21.938843 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:21.938951 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:21.939295 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:21.939301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:21.939415 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 00:48:21.939539 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 00:48:21.939713 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:21.959908 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:41735
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:40109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:22.065130 2325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:22.065359 2325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:22.065470 2325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:22.069321 2325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:22.069371 2325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:22.069423 2325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:48:22.069456 2325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:22.074438 2325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:40109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.249.126:41735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:22.075325 2325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:22.076203 2325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:22.085000 2331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:22.085059 2330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:22.085059 2333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:22.085191 2325 server_base.cc:1034] running on GCE node
I20260925 00:48:22.085711 2325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:22.086295 2325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:22.087476 2325 hybrid_clock.cc:648] HybridClock initialized: now 1790297302087453 us; error 37 us; skew 500 ppm
Sep 25 00:48:22 dist-test-slave-05m0 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:22.090008 2325 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:22.090772 2325 webserver.cc:460] Webserver started at http://127.0.249.66:45629/ using document root <none> and password file <none>
I20260925 00:48:22.091281 2325 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:22.091367 2325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:22.091581 2325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:22.093523 2325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "d858e929a8c34ebdb86ce6e8c5482b8f"
format_stamp: "Formatted at 2026-09-25 00:48:22 on dist-test-slave-05m0"
server_key: "8d5e7e410b988ed295e4091286ac2086"
server_key_iv: "6b82e068e9ea9287107e3bc748c454d3"
server_key_version: "encryptionkey@0"
I20260925 00:48:22.094062 2325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "d858e929a8c34ebdb86ce6e8c5482b8f"
format_stamp: "Formatted at 2026-09-25 00:48:22 on dist-test-slave-05m0"
server_key: "8d5e7e410b988ed295e4091286ac2086"
server_key_iv: "6b82e068e9ea9287107e3bc748c454d3"
server_key_version: "encryptionkey@0"
I20260925 00:48:22.097559 2325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:22.099771 2340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:22.100883 2325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:22.101008 2325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "d858e929a8c34ebdb86ce6e8c5482b8f"
format_stamp: "Formatted at 2026-09-25 00:48:22 on dist-test-slave-05m0"
server_key: "8d5e7e410b988ed295e4091286ac2086"
server_key_iv: "6b82e068e9ea9287107e3bc748c454d3"
server_key_version: "encryptionkey@0"
I20260925 00:48:22.101115 2325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:22.133148 2325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:22.135931 2325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:22.136133 2325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:22.137579 2325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:22.137646 2325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:22.137709 2325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:22.137756 2325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:48:22 dist-test-slave-05m0 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:22.148593 2325 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:37607
I20260925 00:48:22.148603 2455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:37607 every 8 connection(s)
I20260925 00:48:22.149812 2325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:22.151652 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.138346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:59005 (local address 127.0.249.126:41735)
0925 00:48:22.138488 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.138492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.139417 (+ 925us) server_negotiation.cc:407] Connection header received
0925 00:48:22.140698 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.140702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.140769 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.140833 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:22.142682 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.143430 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.144869 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.145114 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.145230 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.147826 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:22.147834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:22.147836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:22.147860 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:22.149622 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.150182 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.150189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.150193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.150292 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.150677 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.150679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.150680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.150924 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:22.151021 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.151426 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.151526 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:48:22.152422 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.138771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41735 (local address 127.0.249.66:59005)
0925 00:48:22.139234 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.139293 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.140416 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.141299 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.141310 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.141788 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:22.142509 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.142533 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.143564 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.143568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.144746 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.144756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.145282 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.145286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.145520 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.146119 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:22.146134 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:22.147663 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:22.149742 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.149746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.149756 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.150060 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.150401 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.150406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.150408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.150577 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.151021 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:22.151026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:22.151228 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.151375 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.151606 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
I20260925 00:48:22.153623 2456 heartbeater.cc:347] Connected to a master server at 127.0.249.126:41735
I20260925 00:48:22.153767 2456 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:22.154284 2456 heartbeater.cc:511] Master 127.0.249.126:41735 requested a full tablet report, sending...
I20260925 00:48:22.155275 2111 ts_manager.cc:194] Registered new tserver with Master: d858e929a8c34ebdb86ce6e8c5482b8f (127.0.249.66:37607)
I20260925 00:48:22.155705 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 2325
I20260925 00:48:22.155819 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:22.156095 997 external_mini_cluster.cc:1428] Setting key a774546b21b2a4f8bfce2338ac860aac
I20260925 00:48:22.156075 2111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:59005
I20260925 00:48:22.164655 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.155552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41735 (local address 127.0.249.66:49029)
0925 00:48:22.157577 (+ 2025us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.157591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.157685 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.157944 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.157947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.158146 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:22.158273 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.158281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.159164 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.159170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.159762 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.159770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.160340 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.160342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.160406 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.160915 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:22.160926 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:22.161227 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:22.163201 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.163204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.163205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.163384 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.163683 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.163685 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.163686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.163755 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.164251 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:22.164253 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:22.164319 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.164395 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.164492 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1946,"thread_start_us":77,"threads_started":1}
I20260925 00:48:22.164808 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.156351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:49029 (local address 127.0.249.126:41735)
0925 00:48:22.156644 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.156648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.157670 (+ 1022us) server_negotiation.cc:407] Connection header received
0925 00:48:22.157768 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.157772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.157821 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.157904 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:22.158396 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.159015 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.159872 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.160243 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.160324 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.161363 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:22.161369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:22.161375 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:22.161429 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:22.163093 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.163482 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.163488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.163492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.163587 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.163842 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.163846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.163851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.164156 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:22.164261 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.164483 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.164619 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:22.182605 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:41735
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:40109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:22.295228 2465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:22.295475 2465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:22.295518 2465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:22.299639 2465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:22.299698 2465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:22.299737 2465 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:48:22.299769 2465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:22.304955 2465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:40109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.249.126:41735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:22.306041 2465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:22.306979 2465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:22.316227 2471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:22.316251 2470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:22.316227 2473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:22.316548 2465 server_base.cc:1034] running on GCE node
I20260925 00:48:22.317083 2465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:22.317785 2465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:22.318943 2465 hybrid_clock.cc:648] HybridClock initialized: now 1790297302318929 us; error 25 us; skew 500 ppm
Sep 25 00:48:22 dist-test-slave-05m0 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:22.321831 2465 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:22.322589 2465 webserver.cc:460] Webserver started at http://127.0.249.67:36127/ using document root <none> and password file <none>
I20260925 00:48:22.323079 2465 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:22.323133 2465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:22.323288 2465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:22.325312 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "6ca18b6fd1c1478995d739fb93667bfc"
format_stamp: "Formatted at 2026-09-25 00:48:22 on dist-test-slave-05m0"
server_key: "5492f258520ff58bf26410d52054ec92"
server_key_iv: "99d749a5cefd9fabbcdbf75201c60b43"
server_key_version: "encryptionkey@0"
I20260925 00:48:22.325874 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "6ca18b6fd1c1478995d739fb93667bfc"
format_stamp: "Formatted at 2026-09-25 00:48:22 on dist-test-slave-05m0"
server_key: "5492f258520ff58bf26410d52054ec92"
server_key_iv: "99d749a5cefd9fabbcdbf75201c60b43"
server_key_version: "encryptionkey@0"
I20260925 00:48:22.329736 2465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:48:22.331980 2480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:22.333139 2465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:22.333267 2465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "6ca18b6fd1c1478995d739fb93667bfc"
format_stamp: "Formatted at 2026-09-25 00:48:22 on dist-test-slave-05m0"
server_key: "5492f258520ff58bf26410d52054ec92"
server_key_iv: "99d749a5cefd9fabbcdbf75201c60b43"
server_key_version: "encryptionkey@0"
I20260925 00:48:22.333375 2465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:22.357100 2465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:22.360615 2465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:22.360833 2465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:22.362733 2465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:22.362810 2465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:22.362879 2465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:22.362910 2465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:48:22 dist-test-slave-05m0 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:22.373306 2465 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:45311
I20260925 00:48:22.373493 2595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:45311 every 8 connection(s)
I20260925 00:48:22.374313 2465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:22.377454 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.363404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:34815 (local address 127.0.249.126:41735)
0925 00:48:22.363572 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.363577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.364403 (+ 826us) server_negotiation.cc:407] Connection header received
0925 00:48:22.365496 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.365500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.365578 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.365644 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:22.367418 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.368304 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.370214 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.370497 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.370583 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.373765 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:22.373769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:22.373771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:22.373796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:22.375486 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.376077 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.376079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.376081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.376157 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.376486 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.376488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.376489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.376659 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:22.376731 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.377197 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.377291 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 00:48:22.378201 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 2465
I20260925 00:48:22.378342 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:22.378382 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.363794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41735 (local address 127.0.249.67:34815)
0925 00:48:22.364230 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.364279 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.365232 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.366105 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.366129 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.366587 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:22.367253 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.367275 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.368448 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.368454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.370078 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.370092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.370673 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.370678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.370898 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.371668 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:22.371691 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:22.373601 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:22.375609 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.375617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.375631 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.375968 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.376238 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.376240 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.376242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.376400 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.376747 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:22.376752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:22.376990 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.377144 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.377465 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
I20260925 00:48:22.378693 997 external_mini_cluster.cc:1428] Setting key 7eb8d8727825dfa1d84e3aff0a7ec6b8
I20260925 00:48:22.379881 2596 heartbeater.cc:347] Connected to a master server at 127.0.249.126:41735
I20260925 00:48:22.380079 2596 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:22.380678 2596 heartbeater.cc:511] Master 127.0.249.126:41735 requested a full tablet report, sending...
I20260925 00:48:22.381744 2111 ts_manager.cc:194] Registered new tserver with Master: 6ca18b6fd1c1478995d739fb93667bfc (127.0.249.67:45311)
I20260925 00:48:22.382570 2111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:34815
I20260925 00:48:22.388535 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.382168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:59339 (local address 127.0.249.126:41735)
0925 00:48:22.382293 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.382298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.382429 (+ 131us) server_negotiation.cc:407] Connection header received
0925 00:48:22.382503 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.382506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.382548 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.382638 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:22.383055 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.383608 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.384663 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.384878 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.384960 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.386002 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:22.386005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:22.386007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:22.386029 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:22.387249 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.387623 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.387626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.387627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.387698 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.387921 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.387923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.387925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.388077 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:22.388153 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.388329 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.388431 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:48:22.388551 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.382055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41735 (local address 127.0.249.67:59339)
0925 00:48:22.382345 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.382366 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.382442 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.382640 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.382642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.382855 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:22.382948 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.382954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.383732 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.383736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.384522 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.384544 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.384974 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.384976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.385042 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.385567 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:22.385578 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:22.385883 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:22.387344 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.387346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.387347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.387526 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.387774 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.387776 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.387777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.387847 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.388155 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:22.388157 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:22.388223 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.388298 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.388395 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":107,"threads_started":1}
I20260925 00:48:22.393114 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 00:48:22 dist-test-slave-05m0 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:48:22 dist-test-slave-05m0 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:22.418340 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.409677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.0.249.126:41735)
0925 00:48:22.409844 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.409849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.409963 (+ 114us) server_negotiation.cc:407] Connection header received
0925 00:48:22.410142 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.410151 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.410196 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.410320 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:22.410983 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.411502 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.412370 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.412597 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.412701 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.415101 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:22.415110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:22.415112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:22.415133 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:22.416716 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.417116 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.417120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.417122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.417200 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.417490 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.417494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.417495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.417681 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:22.417824 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.418059 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.418174 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 00:48:22.421643 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:22.424733 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:22.440053 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.433680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:45311 (local address 127.0.0.1:41646)
0925 00:48:22.434085 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.434138 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.434264 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.435561 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.435567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.435603 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:22.435923 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.435931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.437031 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.437037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.438446 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.438460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.439493 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.439496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.439619 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.439656 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.439762 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.439879 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":194,"spinlock_wait_cycles":5632,"thread_start_us":114,"threads_started":1}
I20260925 00:48:22.440236 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.434456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:37607 (local address 127.0.0.1:52178)
0925 00:48:22.434796 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.434810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.434888 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.435663 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.435665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.435681 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:22.435882 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.435891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.437321 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.437323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.438794 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.438802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.439825 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.439828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.439912 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.439945 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.440013 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.440083 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":194,"spinlock_wait_cycles":1280,"thread_start_us":87,"threads_started":1}
I20260925 00:48:22.440274 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.433707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:41613 (local address 127.0.0.1:44148)
0925 00:48:22.434501 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.434518 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.434599 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.435561 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.435567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.435604 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:22.435880 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.435891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.437307 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.437312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.438763 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.438773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.439783 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.439786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.439871 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.439891 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.439970 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.440049 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":219,"spinlock_wait_cycles":11392,"thread_start_us":110,"threads_started":1}
I20260925 00:48:22.440515 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.433774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.0.249.67:45311)
0925 00:48:22.435111 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.435122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.435141 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:48:22.435213 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.435218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.435423 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.435570 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:22.436046 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.436925 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.438576 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.439360 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.439471 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.439633 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.440250 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.440357 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":125,"threads_started":1}
I20260925 00:48:22.440747 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.434631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.0.249.66:37607)
0925 00:48:22.435101 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.435114 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.435138 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:48:22.435213 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.435218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.435414 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.435570 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:22.436023 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.437207 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.438911 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.439496 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.439603 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.439766 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.440508 (+ 742us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.440609 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":109,"threads_started":1}
I20260925 00:48:22.441780 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.434528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.0.249.65:41613)
0925 00:48:22.435055 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.435062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.435087 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:48:22.435166 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.435171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.435390 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.435836 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:22.435982 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.437168 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.438897 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.439667 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.439770 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.439892 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.440000 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.440095 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":63,"threads_started":1}
I20260925 00:48:22.443579 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7fc13fdb3543a8a84d684180d66eda (DEFAULT_TABLE table=test-table [id=5e8253f6d0b14587b84d0d76ba477dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:22.443579 2530 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7fc13fdb3543a8a84d684180d66eda (DEFAULT_TABLE table=test-table [id=5e8253f6d0b14587b84d0d76ba477dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:22.443876 2250 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7fc13fdb3543a8a84d684180d66eda (DEFAULT_TABLE table=test-table [id=5e8253f6d0b14587b84d0d76ba477dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:22.444720 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7fc13fdb3543a8a84d684180d66eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:22.444722 2530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7fc13fdb3543a8a84d684180d66eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:22.444814 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7fc13fdb3543a8a84d684180d66eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:22.451753 2620 tablet_bootstrap.cc:492] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: Bootstrap starting.
I20260925 00:48:22.452378 2621 tablet_bootstrap.cc:492] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: Bootstrap starting.
I20260925 00:48:22.452781 2622 tablet_bootstrap.cc:492] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: Bootstrap starting.
I20260925 00:48:22.454236 2620 tablet_bootstrap.cc:654] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:22.455018 2621 tablet_bootstrap.cc:654] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:22.455102 2620 log.cc:824] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:22.455219 2622 tablet_bootstrap.cc:654] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:22.455749 2621 log.cc:824] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:22.456023 2622 log.cc:824] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:22.457181 2620 tablet_bootstrap.cc:492] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: No bootstrap required, opened a new log
I20260925 00:48:22.457376 2620 ts_tablet_manager.cc:1397] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:48:22.457872 2621 tablet_bootstrap.cc:492] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: No bootstrap required, opened a new log
I20260925 00:48:22.458051 2622 tablet_bootstrap.cc:492] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: No bootstrap required, opened a new log
I20260925 00:48:22.458074 2621 ts_tablet_manager.cc:1397] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:48:22.458252 2622 ts_tablet_manager.cc:1397] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:48:22.461025 2622 raft_consensus.cc:348] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } }
I20260925 00:48:22.461261 2622 raft_consensus.cc:374] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:22.461314 2622 raft_consensus.cc:729] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca18b6fd1c1478995d739fb93667bfc, State: Initialized, Role: FOLLOWER
I20260925 00:48:22.461797 2622 consensus_queue.cc:260] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } }
I20260925 00:48:22.461761 2621 raft_consensus.cc:348] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } }
I20260925 00:48:22.461760 2620 raft_consensus.cc:348] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } }
I20260925 00:48:22.461994 2620 raft_consensus.cc:374] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:22.461994 2621 raft_consensus.cc:374] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:22.462054 2621 raft_consensus.cc:729] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d858e929a8c34ebdb86ce6e8c5482b8f, State: Initialized, Role: FOLLOWER
I20260925 00:48:22.462062 2620 raft_consensus.cc:729] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa23292df91b4f0092147676cf979b3f, State: Initialized, Role: FOLLOWER
I20260925 00:48:22.462481 2596 heartbeater.cc:503] Master 127.0.249.126:41735 was elected leader, sending a full tablet report...
I20260925 00:48:22.462503 2620 consensus_queue.cc:260] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } }
I20260925 00:48:22.462890 2622 ts_tablet_manager.cc:1428] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:48:22.463387 2620 ts_tablet_manager.cc:1428] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260925 00:48:22.463500 2316 heartbeater.cc:503] Master 127.0.249.126:41735 was elected leader, sending a full tablet report...
I20260925 00:48:22.463768 2621 consensus_queue.cc:260] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } }
I20260925 00:48:22.464396 2456 heartbeater.cc:503] Master 127.0.249.126:41735 was elected leader, sending a full tablet report...
I20260925 00:48:22.464689 2621 ts_tablet_manager.cc:1428] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 00:48:22.614296 2627 raft_consensus.cc:482] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:22.614509 2627 raft_consensus.cc:504] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } }
I20260925 00:48:22.615813 2627 leader_election.cc:288] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ca18b6fd1c1478995d739fb93667bfc (127.0.249.67:45311), d858e929a8c34ebdb86ce6e8c5482b8f (127.0.249.66:37607)
I20260925 00:48:22.619908 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.616150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:37663 (local address 127.0.249.66:37607)
0925 00:48:22.616274 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.616279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.616531 (+ 252us) server_negotiation.cc:407] Connection header received
0925 00:48:22.616789 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.616792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.616848 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.616933 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:22.617253 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.617784 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.618902 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.619300 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.619447 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.619503 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.619702 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.619773 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:48:22.619923 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.616156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:55315 (local address 127.0.249.67:45311)
0925 00:48:22.616274 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.616279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.616543 (+ 264us) server_negotiation.cc:407] Connection header received
0925 00:48:22.616734 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.616737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.616791 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.616847 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:22.617199 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.617767 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.618901 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.619308 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.619439 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.619493 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.619740 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.619810 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 00:48:22.619953 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.616046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:37607 (local address 127.0.249.65:37663)
0925 00:48:22.616415 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.616464 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.616612 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.616952 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.616954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.616969 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:22.617104 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.617109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.617932 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.617935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.618782 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.618790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.619439 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.619442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.619542 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.619604 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.619662 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.619748 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":121,"spinlock_wait_cycles":11008,"thread_start_us":128,"threads_started":1}
I20260925 00:48:22.619959 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.616067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:45311 (local address 127.0.249.65:55315)
0925 00:48:22.616434 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.616445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.616609 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.616866 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.616871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.616892 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:22.617078 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.617085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.617873 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.617896 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.618777 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.618786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.619447 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.619451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.619577 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.619621 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.619715 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.619776 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":41,"spinlock_wait_cycles":26240,"thread_start_us":110,"threads_started":1}
I20260925 00:48:22.620576 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fc13fdb3543a8a84d684180d66eda" candidate_uuid: "aa23292df91b4f0092147676cf979b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" is_pre_election: true
I20260925 00:48:22.620628 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fc13fdb3543a8a84d684180d66eda" candidate_uuid: "aa23292df91b4f0092147676cf979b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca18b6fd1c1478995d739fb93667bfc" is_pre_election: true
I20260925 00:48:22.620872 2410 raft_consensus.cc:2454] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa23292df91b4f0092147676cf979b3f in term 0.
I20260925 00:48:22.620893 2550 raft_consensus.cc:2454] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa23292df91b4f0092147676cf979b3f in term 0.
I20260925 00:48:22.621358 2202 leader_election.cc:302] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa23292df91b4f0092147676cf979b3f, d858e929a8c34ebdb86ce6e8c5482b8f; no voters:
I20260925 00:48:22.621690 2627 raft_consensus.cc:2781] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:22.621776 2627 raft_consensus.cc:482] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:22.621819 2627 raft_consensus.cc:3037] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:22.622803 2627 raft_consensus.cc:504] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } }
I20260925 00:48:22.623117 2627 leader_election.cc:288] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [CANDIDATE]: Term 1 election: Requested vote from peers 6ca18b6fd1c1478995d739fb93667bfc (127.0.249.67:45311), d858e929a8c34ebdb86ce6e8c5482b8f (127.0.249.66:37607)
I20260925 00:48:22.623476 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fc13fdb3543a8a84d684180d66eda" candidate_uuid: "aa23292df91b4f0092147676cf979b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f"
I20260925 00:48:22.623534 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fc13fdb3543a8a84d684180d66eda" candidate_uuid: "aa23292df91b4f0092147676cf979b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca18b6fd1c1478995d739fb93667bfc"
I20260925 00:48:22.623641 2410 raft_consensus.cc:3037] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:22.623689 2550 raft_consensus.cc:3037] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:22.624805 2410 raft_consensus.cc:2454] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa23292df91b4f0092147676cf979b3f in term 1.
I20260925 00:48:22.624946 2550 raft_consensus.cc:2454] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa23292df91b4f0092147676cf979b3f in term 1.
W20260925 00:48:22.625283 2597 tablet.cc:2369] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:22.625427 2202 leader_election.cc:302] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa23292df91b4f0092147676cf979b3f, d858e929a8c34ebdb86ce6e8c5482b8f; no voters:
I20260925 00:48:22.625702 2627 raft_consensus.cc:2781] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:22.625921 2627 raft_consensus.cc:686] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 1 LEADER]: Becoming Leader. State: Replica: aa23292df91b4f0092147676cf979b3f, State: Running, Role: LEADER
I20260925 00:48:22.626274 2627 consensus_queue.cc:237] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } }
I20260925 00:48:22.629293 2110 catalog_manager.cc:5495] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f reported cstate change: term changed from 0 to 1, leader changed from <none> to aa23292df91b4f0092147676cf979b3f (127.0.249.65). New cstate: current_term: 1 leader_uuid: "aa23292df91b4f0092147676cf979b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } health_report { overall_health: UNKNOWN } } }
W20260925 00:48:22.650944 2457 tablet.cc:2369] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:48:22.675729 2317 tablet.cc:2369] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:22.684270 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.680815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.0.249.65:41613)
0925 00:48:22.680982 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.680985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.680999 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:22.681180 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.681182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.681234 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.681314 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:22.681570 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.682119 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.682950 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.683189 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.683269 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.683541 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:22.683899 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:22.684022 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.684115 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.684167 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:48:22.689595 2410 raft_consensus.cc:1261] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 FOLLOWER]: Refusing update from remote peer aa23292df91b4f0092147676cf979b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:22.689695 2550 raft_consensus.cc:1261] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Refusing update from remote peer aa23292df91b4f0092147676cf979b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:22.690295 2627 consensus_queue.cc:1035] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:22.690517 2631 consensus_queue.cc:1035] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:22.697758 2636 mvcc.cc:205] Tried to move back new op lower bound from 7333057751810170880 to 7333057751559901184. Current Snapshot: MvccSnapshot[applied={T|T < 7333057751810170880}]
I20260925 00:48:22.707866 2110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:22.708014 2110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:22.710462 2110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976: Sending DeleteTablet for 3 replicas of tablet 8c7fc13fdb3543a8a84d684180d66eda
I20260925 00:48:22.711217 2250 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7fc13fdb3543a8a84d684180d66eda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:22 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:44148
I20260925 00:48:22.711256 2530 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7fc13fdb3543a8a84d684180d66eda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:22 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:41646
I20260925 00:48:22.711354 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7fc13fdb3543a8a84d684180d66eda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:22 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:52178
I20260925 00:48:22.711711 2644 tablet_replica.cc:330] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: stopping tablet replica
I20260925 00:48:22.711825 2645 tablet_replica.cc:330] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: stopping tablet replica
I20260925 00:48:22.712105 2645 raft_consensus.cc:2229] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:22.712105 2644 raft_consensus.cc:2229] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:22.712420 2645 raft_consensus.cc:2258] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:22.712620 2644 raft_consensus.cc:2258] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:22.712559 2110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 00:48:22.712782 2646 tablet_replica.cc:330] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: stopping tablet replica
W20260925 00:48:22.712970 2110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:22.713264 2646 raft_consensus.cc:2229] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:22.713572 2646 raft_consensus.cc:2258] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:22.715621 2645 ts_tablet_manager.cc:1905] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:22.718042 2250 tablet_service.cc:1467] Processing CreateTablet for tablet 12c8deab13684779a7e27fc31a2c067b (DEFAULT_TABLE table=test-table [id=e2330820183c46418059f5c1b2ddb2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:22.718324 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c8deab13684779a7e27fc31a2c067b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:22.718842 2644 ts_tablet_manager.cc:1905] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:22.720881 2620 tablet_bootstrap.cc:492] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: Bootstrap starting.
I20260925 00:48:22.721205 2644 ts_tablet_manager.cc:1918] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:22.721299 2644 log.cc:1196] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/8c7fc13fdb3543a8a84d684180d66eda
I20260925 00:48:22.721650 2644 ts_tablet_manager.cc:1939] T 8c7fc13fdb3543a8a84d684180d66eda P aa23292df91b4f0092147676cf979b3f: Deleting consensus metadata
I20260925 00:48:22.721809 2645 ts_tablet_manager.cc:1918] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:22.721956 2620 tablet_bootstrap.cc:654] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:22.722061 2645 log.cc:1196] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/8c7fc13fdb3543a8a84d684180d66eda
I20260925 00:48:22.722431 2645 ts_tablet_manager.cc:1939] T 8c7fc13fdb3543a8a84d684180d66eda P 6ca18b6fd1c1478995d739fb93667bfc: Deleting consensus metadata
I20260925 00:48:22.722882 2097 catalog_manager.cc:4841] TS aa23292df91b4f0092147676cf979b3f (127.0.249.65:41613): tablet 8c7fc13fdb3543a8a84d684180d66eda (table test-table [id=5e8253f6d0b14587b84d0d76ba477dce]) successfully deleted
I20260925 00:48:22.722923 2530 tablet_service.cc:1467] Processing CreateTablet for tablet 12c8deab13684779a7e27fc31a2c067b (DEFAULT_TABLE table=test-table [id=e2330820183c46418059f5c1b2ddb2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:22.722924 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 12c8deab13684779a7e27fc31a2c067b (DEFAULT_TABLE table=test-table [id=e2330820183c46418059f5c1b2ddb2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:22.723156 2530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c8deab13684779a7e27fc31a2c067b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:22.723156 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c8deab13684779a7e27fc31a2c067b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:22.725239 2622 tablet_bootstrap.cc:492] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: Bootstrap starting.
I20260925 00:48:22.725448 2621 tablet_bootstrap.cc:492] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: Bootstrap starting.
I20260925 00:48:22.725883 2099 catalog_manager.cc:4841] TS 6ca18b6fd1c1478995d739fb93667bfc (127.0.249.67:45311): tablet 8c7fc13fdb3543a8a84d684180d66eda (table test-table [id=5e8253f6d0b14587b84d0d76ba477dce]) successfully deleted
I20260925 00:48:22.726341 2621 tablet_bootstrap.cc:654] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:22.726383 2622 tablet_bootstrap.cc:654] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:22.727020 2646 ts_tablet_manager.cc:1905] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:22.727384 2621 tablet_bootstrap.cc:492] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: No bootstrap required, opened a new log
I20260925 00:48:22.727459 2621 ts_tablet_manager.cc:1397] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 00:48:22.728154 2621 raft_consensus.cc:348] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.728286 2621 raft_consensus.cc:374] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:22.728334 2621 raft_consensus.cc:729] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d858e929a8c34ebdb86ce6e8c5482b8f, State: Initialized, Role: FOLLOWER
I20260925 00:48:22.728462 2621 consensus_queue.cc:260] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.728763 2621 ts_tablet_manager.cc:1428] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 00:48:22.729456 2646 ts_tablet_manager.cc:1918] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:22.729597 2646 log.cc:1196] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/8c7fc13fdb3543a8a84d684180d66eda
I20260925 00:48:22.729893 2646 ts_tablet_manager.cc:1939] T 8c7fc13fdb3543a8a84d684180d66eda P d858e929a8c34ebdb86ce6e8c5482b8f: Deleting consensus metadata
I20260925 00:48:22.730954 2098 catalog_manager.cc:4841] TS d858e929a8c34ebdb86ce6e8c5482b8f (127.0.249.66:37607): tablet 8c7fc13fdb3543a8a84d684180d66eda (table test-table [id=5e8253f6d0b14587b84d0d76ba477dce]) successfully deleted
I20260925 00:48:22.731519 2620 tablet_bootstrap.cc:492] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: No bootstrap required, opened a new log
I20260925 00:48:22.731627 2620 ts_tablet_manager.cc:1397] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20260925 00:48:22.732002 2620 raft_consensus.cc:348] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.732093 2620 raft_consensus.cc:374] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:22.732115 2620 raft_consensus.cc:729] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa23292df91b4f0092147676cf979b3f, State: Initialized, Role: FOLLOWER
I20260925 00:48:22.732239 2620 consensus_queue.cc:260] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.732507 2620 ts_tablet_manager.cc:1428] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:48:22.737131 2622 tablet_bootstrap.cc:492] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: No bootstrap required, opened a new log
I20260925 00:48:22.737228 2622 ts_tablet_manager.cc:1397] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260925 00:48:22.737730 2622 raft_consensus.cc:348] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.737849 2622 raft_consensus.cc:374] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:22.737887 2622 raft_consensus.cc:729] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca18b6fd1c1478995d739fb93667bfc, State: Initialized, Role: FOLLOWER
I20260925 00:48:22.738013 2622 consensus_queue.cc:260] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.738332 2622 ts_tablet_manager.cc:1428] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 00:48:22.755220 2628 raft_consensus.cc:482] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:22.755362 2628 raft_consensus.cc:504] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.756402 2628 leader_election.cc:288] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa23292df91b4f0092147676cf979b3f (127.0.249.65:41613), 6ca18b6fd1c1478995d739fb93667bfc (127.0.249.67:45311)
I20260925 00:48:22.761222 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.756728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:37015 (local address 127.0.249.67:45311)
0925 00:48:22.756851 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.756855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.757357 (+ 502us) server_negotiation.cc:407] Connection header received
0925 00:48:22.757522 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.757525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.757584 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.757631 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:22.758060 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.758761 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.760174 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.760655 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.760793 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.760850 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.761037 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.761104 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:48:22.761250 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.756631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:45311 (local address 127.0.249.66:37015)
0925 00:48:22.757267 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.757278 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.757389 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.757788 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.757792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.757811 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:22.757925 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.757931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.758894 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.758897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.760041 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.760052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.760793 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.760797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.760902 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.760931 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.761016 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.761089 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":14,"thread_start_us":117,"threads_started":1}
I20260925 00:48:22.761472 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.756955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:57263 (local address 127.0.249.65:41613)
0925 00:48:22.757095 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.757099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.757113 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:22.757277 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.757280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.757330 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.757440 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:22.757728 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.758506 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.760118 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.760762 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.760875 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.760925 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.761233 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.761304 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:48:22.761868 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.756603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:41613 (local address 127.0.249.66:57263)
0925 00:48:22.757012 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.757027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.757143 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.757452 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.757455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.757472 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:22.757614 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.757624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.758621 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.758624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.759981 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.759991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.760986 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.760991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.761098 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.761120 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.761657 (+ 537us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.761733 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":51,"thread_start_us":75,"threads_started":1}
I20260925 00:48:22.761968 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c8deab13684779a7e27fc31a2c067b" candidate_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca18b6fd1c1478995d739fb93667bfc" is_pre_election: true
I20260925 00:48:22.762121 2550 raft_consensus.cc:2454] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d858e929a8c34ebdb86ce6e8c5482b8f in term 0.
I20260925 00:48:22.762358 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c8deab13684779a7e27fc31a2c067b" candidate_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa23292df91b4f0092147676cf979b3f" is_pre_election: true
I20260925 00:48:22.762477 2344 leader_election.cc:302] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ca18b6fd1c1478995d739fb93667bfc, d858e929a8c34ebdb86ce6e8c5482b8f; no voters:
I20260925 00:48:22.762609 2270 raft_consensus.cc:2454] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d858e929a8c34ebdb86ce6e8c5482b8f in term 0.
I20260925 00:48:22.762774 2628 raft_consensus.cc:2781] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:22.762836 2628 raft_consensus.cc:482] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:22.762876 2628 raft_consensus.cc:3037] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:22.763820 2628 raft_consensus.cc:504] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.764179 2628 leader_election.cc:288] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [CANDIDATE]: Term 1 election: Requested vote from peers aa23292df91b4f0092147676cf979b3f (127.0.249.65:41613), 6ca18b6fd1c1478995d739fb93667bfc (127.0.249.67:45311)
I20260925 00:48:22.764500 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c8deab13684779a7e27fc31a2c067b" candidate_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa23292df91b4f0092147676cf979b3f"
I20260925 00:48:22.764508 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c8deab13684779a7e27fc31a2c067b" candidate_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca18b6fd1c1478995d739fb93667bfc"
I20260925 00:48:22.764621 2550 raft_consensus.cc:3037] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:22.764650 2270 raft_consensus.cc:3037] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:22.765645 2270 raft_consensus.cc:2454] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d858e929a8c34ebdb86ce6e8c5482b8f in term 1.
I20260925 00:48:22.765750 2550 raft_consensus.cc:2454] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d858e929a8c34ebdb86ce6e8c5482b8f in term 1.
I20260925 00:48:22.765919 2342 leader_election.cc:302] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa23292df91b4f0092147676cf979b3f, d858e929a8c34ebdb86ce6e8c5482b8f; no voters:
I20260925 00:48:22.766165 2628 raft_consensus.cc:2781] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:22.766392 2628 raft_consensus.cc:686] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 LEADER]: Becoming Leader. State: Replica: d858e929a8c34ebdb86ce6e8c5482b8f, State: Running, Role: LEADER
I20260925 00:48:22.766650 2628 consensus_queue.cc:237] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.768934 2110 catalog_manager.cc:5495] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f reported cstate change: term changed from 0 to 1, leader changed from <none> to d858e929a8c34ebdb86ce6e8c5482b8f (127.0.249.66). New cstate: current_term: 1 leader_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } health_report { overall_health: UNKNOWN } } }
Sep 25 00:48:22 dist-test-slave-05m0 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:22.819794 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.811060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.0.249.126:41735)
0925 00:48:22.811185 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:22.811192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:22.811359 (+ 167us) server_negotiation.cc:407] Connection header received
0925 00:48:22.811513 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:22.811516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:22.811563 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:22.811647 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:22.812267 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.812782 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.813641 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.813847 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.813951 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.816628 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:22.816632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:22.816637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:22.816662 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:22.818111 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.818577 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.818581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.818583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.818656 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:22.818929 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:22.818932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:22.818933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:22.819112 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:22.819218 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:22.819496 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 00:48:22.819657 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":42}
I20260925 00:48:22.819808 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:22.810981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41735 (local address 127.0.0.1:59288)
0925 00:48:22.811272 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:22.811284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:22.811420 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:22.811664 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:22.811667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:22.812065 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:22.812152 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.812159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.812886 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.812892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.813534 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:22.813544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:22.813954 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:22.813957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:22.814044 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:22.814706 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:22.814725 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:22.816490 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:22.818217 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.818227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.818233 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.818459 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.818751 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:22.818755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:22.818756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:22.818838 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:22.819217 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:22.819224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:22.819332 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 00:48:22.819442 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 00:48:22.819604 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":107,"threads_started":1}
I20260925 00:48:22.824158 2110 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:59288:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 00:48:22.824887 2110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:22.829746 2250 tablet_service.cc:1467] Processing CreateTablet for tablet 4292889598914ce9bd2c765823a7f406 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d91b375ca98e4fdc953fa280e820f567]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:48:22.829962 2530 tablet_service.cc:1467] Processing CreateTablet for tablet 4292889598914ce9bd2c765823a7f406 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d91b375ca98e4fdc953fa280e820f567]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:48:22.830137 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4292889598914ce9bd2c765823a7f406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:22.829979 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 4292889598914ce9bd2c765823a7f406 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d91b375ca98e4fdc953fa280e820f567]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:48:22.830294 2530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4292889598914ce9bd2c765823a7f406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:22.830368 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4292889598914ce9bd2c765823a7f406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:22.832917 2620 tablet_bootstrap.cc:492] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: Bootstrap starting.
I20260925 00:48:22.832970 2622 tablet_bootstrap.cc:492] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: Bootstrap starting.
I20260925 00:48:22.833989 2622 tablet_bootstrap.cc:654] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:22.834002 2620 tablet_bootstrap.cc:654] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:22.834661 2621 tablet_bootstrap.cc:492] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: Bootstrap starting.
I20260925 00:48:22.835501 2620 tablet_bootstrap.cc:492] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: No bootstrap required, opened a new log
I20260925 00:48:22.835609 2621 tablet_bootstrap.cc:654] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:22.835606 2620 ts_tablet_manager.cc:1397] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 00:48:22.836076 2620 raft_consensus.cc:348] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.836199 2620 raft_consensus.cc:374] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:22.836230 2620 raft_consensus.cc:729] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa23292df91b4f0092147676cf979b3f, State: Initialized, Role: FOLLOWER
I20260925 00:48:22.836335 2620 consensus_queue.cc:260] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.836625 2620 ts_tablet_manager.cc:1428] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 00:48:22.836584 2631 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:22.836710 2631 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:22.836741 2622 tablet_bootstrap.cc:492] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: No bootstrap required, opened a new log
I20260925 00:48:22.836820 2622 ts_tablet_manager.cc:1397] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 00:48:22.837208 2622 raft_consensus.cc:348] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.837338 2622 raft_consensus.cc:374] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:22.837414 2622 raft_consensus.cc:729] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca18b6fd1c1478995d739fb93667bfc, State: Initialized, Role: FOLLOWER
I20260925 00:48:22.837555 2622 consensus_queue.cc:260] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.837844 2622 ts_tablet_manager.cc:1428] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:22.837862 2626 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:22.837998 2626 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:22.839289 2621 tablet_bootstrap.cc:492] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: No bootstrap required, opened a new log
I20260925 00:48:22.839380 2621 ts_tablet_manager.cc:1397] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 00:48:22.839941 2621 raft_consensus.cc:348] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.840066 2621 raft_consensus.cc:374] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:22.840108 2621 raft_consensus.cc:729] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d858e929a8c34ebdb86ce6e8c5482b8f, State: Initialized, Role: FOLLOWER
I20260925 00:48:22.840246 2621 consensus_queue.cc:260] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:22.840535 2621 ts_tablet_manager.cc:1428] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:48:22.840540 2628 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:22.840695 2628 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:23.026156 2626 raft_consensus.cc:482] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:23.026324 2626 raft_consensus.cc:504] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:23.027432 2626 leader_election.cc:288] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa23292df91b4f0092147676cf979b3f (127.0.249.65:41613), d858e929a8c34ebdb86ce6e8c5482b8f (127.0.249.66:37607)
I20260925 00:48:23.032876 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.027773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:58135 (local address 127.0.249.65:41613)
0925 00:48:23.027970 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:23.027974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:23.028054 (+ 80us) server_negotiation.cc:407] Connection header received
0925 00:48:23.028202 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:23.028205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:23.028254 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:23.028338 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:23.028759 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.029596 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.031477 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.032109 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.032204 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.032255 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:23.032639 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 00:48:23.032744 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 00:48:23.032948 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.027817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:40661 (local address 127.0.249.66:37607)
0925 00:48:23.027935 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:23.027939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:23.028112 (+ 173us) server_negotiation.cc:407] Connection header received
0925 00:48:23.028202 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:23.028205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:23.028254 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:23.028338 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:23.028759 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.029604 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.031477 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.032079 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.032203 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.032255 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:23.032757 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 00:48:23.032830 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:48:23.033270 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.027655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:41613 (local address 127.0.249.67:58135)
0925 00:48:23.027962 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:23.027978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:23.028098 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:23.028364 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:23.028367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:23.028407 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:23.028651 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.028658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.029713 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.029716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.031349 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.031359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.032318 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.032321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.032480 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.032515 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 00:48:23.033029 (+ 514us) client_negotiation.cc:240] Negotiation successful
0925 00:48:23.033117 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":218,"spinlock_wait_cycles":30208,"thread_start_us":71,"threads_started":1}
I20260925 00:48:23.033322 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.027655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:37607 (local address 127.0.249.67:40661)
0925 00:48:23.028014 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:23.028027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:23.028110 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:23.028364 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:23.028367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:23.028408 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:23.028629 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.028638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.029713 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.029716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.031353 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.031359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.032318 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.032321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.032631 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.032653 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:48:23.033146 (+ 493us) client_negotiation.cc:240] Negotiation successful
0925 00:48:23.033203 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":514,"spinlock_wait_cycles":4096,"thread_start_us":76,"threads_started":1}
I20260925 00:48:23.034550 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4292889598914ce9bd2c765823a7f406" candidate_uuid: "6ca18b6fd1c1478995d739fb93667bfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" is_pre_election: true
I20260925 00:48:23.034602 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4292889598914ce9bd2c765823a7f406" candidate_uuid: "6ca18b6fd1c1478995d739fb93667bfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa23292df91b4f0092147676cf979b3f" is_pre_election: true
I20260925 00:48:23.034727 2410 raft_consensus.cc:2454] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ca18b6fd1c1478995d739fb93667bfc in term 0.
I20260925 00:48:23.035282 2483 leader_election.cc:302] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ca18b6fd1c1478995d739fb93667bfc, d858e929a8c34ebdb86ce6e8c5482b8f; no voters:
I20260925 00:48:23.035823 2626 raft_consensus.cc:2781] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:23.035904 2626 raft_consensus.cc:482] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:23.035965 2626 raft_consensus.cc:3037] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:23.037343 2270 raft_consensus.cc:2454] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ca18b6fd1c1478995d739fb93667bfc in term 0.
I20260925 00:48:23.037647 2626 raft_consensus.cc:504] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:23.037987 2626 leader_election.cc:288] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [CANDIDATE]: Term 1 election: Requested vote from peers aa23292df91b4f0092147676cf979b3f (127.0.249.65:41613), d858e929a8c34ebdb86ce6e8c5482b8f (127.0.249.66:37607)
I20260925 00:48:23.038661 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4292889598914ce9bd2c765823a7f406" candidate_uuid: "6ca18b6fd1c1478995d739fb93667bfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f"
I20260925 00:48:23.038830 2410 raft_consensus.cc:3037] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:23.040153 2410 raft_consensus.cc:2454] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ca18b6fd1c1478995d739fb93667bfc in term 1.
I20260925 00:48:23.040448 2483 leader_election.cc:302] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ca18b6fd1c1478995d739fb93667bfc, d858e929a8c34ebdb86ce6e8c5482b8f; no voters:
I20260925 00:48:23.041549 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4292889598914ce9bd2c765823a7f406" candidate_uuid: "6ca18b6fd1c1478995d739fb93667bfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa23292df91b4f0092147676cf979b3f"
I20260925 00:48:23.041679 2270 raft_consensus.cc:3037] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:23.042522 2270 raft_consensus.cc:2454] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ca18b6fd1c1478995d739fb93667bfc in term 1.
I20260925 00:48:23.042908 2626 raft_consensus.cc:2781] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:23.043087 2626 raft_consensus.cc:686] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [term 1 LEADER]: Becoming Leader. State: Replica: 6ca18b6fd1c1478995d739fb93667bfc, State: Running, Role: LEADER
I20260925 00:48:23.043336 2626 consensus_queue.cc:237] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } }
I20260925 00:48:23.044723 2665 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: TxnStatusTablet state changed. Reason: New leader 6ca18b6fd1c1478995d739fb93667bfc. Latest consensus state: current_term: 1 leader_uuid: "6ca18b6fd1c1478995d739fb93667bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:23.044831 2665 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:48:23.045560 2667 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 00:48:23.045708 2667 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 00:48:23.046159 2110 catalog_manager.cc:5495] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ca18b6fd1c1478995d739fb93667bfc (127.0.249.67). New cstate: current_term: 1 leader_uuid: "6ca18b6fd1c1478995d739fb93667bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } health_report { overall_health: HEALTHY } } }
I20260925 00:48:23.082729 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.077320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:45311 (local address 127.0.0.1:41658)
0925 00:48:23.077526 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:23.077547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:23.077703 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:23.078203 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:23.078206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:23.078214 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:23.078319 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.078325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.079462 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.079466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.080572 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.080584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.081279 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.081284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.081445 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.081504 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:23.082383 (+ 879us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:23.082400 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:48:23.082473 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 00:48:23.082571 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 00:48:23.083024 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.077698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.0.249.67:45311)
0925 00:48:23.077813 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:23.077818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:23.077832 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:23.077909 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:23.077912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:23.077978 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:23.078079 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:23.078470 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.079355 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.080722 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.081168 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.081289 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.081676 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:23.082270 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:23.082392 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:23.082490 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 00:48:23.082564 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:48:23.085954 2410 raft_consensus.cc:1261] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 FOLLOWER]: Refusing update from remote peer 6ca18b6fd1c1478995d739fb93667bfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:23.086144 2270 raft_consensus.cc:1261] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f [term 1 FOLLOWER]: Refusing update from remote peer 6ca18b6fd1c1478995d739fb93667bfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:23.086503 2665 consensus_queue.cc:1035] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:23.087118 2665 consensus_queue.cc:1035] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:23.087976 2628 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: TxnStatusTablet state changed. Reason: New leader 6ca18b6fd1c1478995d739fb93667bfc. Latest consensus state: current_term: 1 leader_uuid: "6ca18b6fd1c1478995d739fb93667bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:23.088086 2628 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:23.089542 2665 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ca18b6fd1c1478995d739fb93667bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:23.089649 2665 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:48:23.089598 2631 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: TxnStatusTablet state changed. Reason: New leader 6ca18b6fd1c1478995d739fb93667bfc. Latest consensus state: current_term: 1 leader_uuid: "6ca18b6fd1c1478995d739fb93667bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:23.089710 2631 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:23.090428 2635 mvcc.cc:205] Tried to move back new op lower bound from 7333057753435328512 to 7333057753268678656. Current Snapshot: MvccSnapshot[applied={T|T < 7333057753435328512}]
I20260925 00:48:23.090587 2628 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ca18b6fd1c1478995d739fb93667bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:23.090682 2628 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P d858e929a8c34ebdb86ce6e8c5482b8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:23.095454 2626 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ca18b6fd1c1478995d739fb93667bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:23.095563 2626 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P 6ca18b6fd1c1478995d739fb93667bfc: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:48:23.096103 2631 tablet_replica.cc:439] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ca18b6fd1c1478995d739fb93667bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 } } peers { permanent_uuid: "d858e929a8c34ebdb86ce6e8c5482b8f" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37607 } } peers { permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 } } }
I20260925 00:48:23.096223 2631 tablet_replica.cc:442] T 4292889598914ce9bd2c765823a7f406 P aa23292df91b4f0092147676cf979b3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:23.096657 2634 mvcc.cc:205] Tried to move back new op lower bound from 7333057753435328512 to 7333057753268678656. Current Snapshot: MvccSnapshot[applied={T|T < 7333057753435328512 or (T in {7333057753435328512})}]
I20260925 00:48:23.100654 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.095785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.0.249.66:37607)
0925 00:48:23.095887 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:23.095891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:23.095905 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:23.095973 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:23.095975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:23.096042 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:23.096143 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:23.097088 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.097983 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.099251 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.099475 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.099555 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.099834 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:23.100218 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:23.100303 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:23.100439 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 00:48:23.100484 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:48:23.111756 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.107460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:50699 (local address 127.0.249.67:45311)
0925 00:48:23.107574 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:23.107578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:23.107763 (+ 185us) server_negotiation.cc:407] Connection header received
0925 00:48:23.107856 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:23.107858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:23.107896 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:23.108008 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:23.108284 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.108760 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.110759 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.110979 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.111058 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.111380 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:23.111442 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:23.111494 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:23.111624 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 00:48:23.111663 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:48:23.111858 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.107330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:45311 (local address 127.0.249.66:50699)
0925 00:48:23.107654 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:23.107669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:23.107773 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:23.108017 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:23.108020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:23.108028 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:23.108174 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.108181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.108881 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.108885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.110634 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.110647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.111095 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.111100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.111206 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.111258 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:23.111525 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:23.111533 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 00:48:23.111599 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 00:48:23.111685 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":127,"threads_started":1}
I20260925 00:48:23.118465 2550 raft_consensus.cc:1261] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Refusing update from remote peer d858e929a8c34ebdb86ce6e8c5482b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:23.118484 2270 raft_consensus.cc:1261] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [term 1 FOLLOWER]: Refusing update from remote peer d858e929a8c34ebdb86ce6e8c5482b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:23.118885 2628 consensus_queue.cc:1035] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca18b6fd1c1478995d739fb93667bfc" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:23.119282 2652 consensus_queue.cc:1035] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa23292df91b4f0092147676cf979b3f" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 41613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:23.143158 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.139263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:43777 (local address 127.0.249.66:37607)
0925 00:48:23.139391 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:23.139396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:23.139586 (+ 190us) server_negotiation.cc:407] Connection header received
0925 00:48:23.139727 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:23.139730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:23.139768 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:23.139863 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:23.140228 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.140970 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.142068 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.142280 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.142427 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.142710 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:23.142821 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:23.142885 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:23.142980 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 00:48:23.143025 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:23.143183 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.139203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:37607 (local address 127.0.249.67:43777)
0925 00:48:23.139515 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:23.139531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:23.139634 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:23.139997 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:23.140000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:23.140008 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:23.140123 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.140128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.141091 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.141097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.141953 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:23.141963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.142426 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.142430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:23.142527 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.142576 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:23.142889 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:23.142898 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 00:48:23.142971 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 00:48:23.143045 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
I20260925 00:48:23.169574 2112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:23.169760 2112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:23.171912 2112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 493dd2460b6140acb4e47a8936123976: Sending DeleteTablet for 3 replicas of tablet 12c8deab13684779a7e27fc31a2c067b
I20260925 00:48:23.172446 2250 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c8deab13684779a7e27fc31a2c067b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:23 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:44148
I20260925 00:48:23.172652 2644 tablet_replica.cc:330] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: stopping tablet replica
I20260925 00:48:23.172704 2530 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c8deab13684779a7e27fc31a2c067b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:23 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:41646
I20260925 00:48:23.172784 2644 raft_consensus.cc:2229] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:23.172888 2645 tablet_replica.cc:330] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: stopping tablet replica
I20260925 00:48:23.172919 2644 raft_consensus.cc:2258] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:23.173008 2645 raft_consensus.cc:2229] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:23.173131 2645 raft_consensus.cc:2258] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:23.173372 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c8deab13684779a7e27fc31a2c067b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:23 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:52178
I20260925 00:48:23.173614 2646 tablet_replica.cc:330] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: stopping tablet replica
I20260925 00:48:23.173722 2646 raft_consensus.cc:2229] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:23.173871 2644 ts_tablet_manager.cc:1905] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:23.173986 2646 raft_consensus.cc:2258] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:23.174234 2645 ts_tablet_manager.cc:1905] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:23.174670 2646 ts_tablet_manager.cc:1905] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:23.176719 2644 ts_tablet_manager.cc:1918] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:23.176795 2644 log.cc:1196] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/12c8deab13684779a7e27fc31a2c067b
I20260925 00:48:23.176923 2645 ts_tablet_manager.cc:1918] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:23.176990 2645 log.cc:1196] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/12c8deab13684779a7e27fc31a2c067b
I20260925 00:48:23.177074 2644 ts_tablet_manager.cc:1939] T 12c8deab13684779a7e27fc31a2c067b P aa23292df91b4f0092147676cf979b3f: Deleting consensus metadata
I20260925 00:48:23.177230 2645 ts_tablet_manager.cc:1939] T 12c8deab13684779a7e27fc31a2c067b P 6ca18b6fd1c1478995d739fb93667bfc: Deleting consensus metadata
I20260925 00:48:23.177309 2646 ts_tablet_manager.cc:1918] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:23.177407 2646 log.cc:1196] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/12c8deab13684779a7e27fc31a2c067b
I20260925 00:48:23.177673 2646 ts_tablet_manager.cc:1939] T 12c8deab13684779a7e27fc31a2c067b P d858e929a8c34ebdb86ce6e8c5482b8f: Deleting consensus metadata
I20260925 00:48:23.177743 2097 catalog_manager.cc:4841] TS aa23292df91b4f0092147676cf979b3f (127.0.249.65:41613): tablet 12c8deab13684779a7e27fc31a2c067b (table test-table [id=e2330820183c46418059f5c1b2ddb2ea]) successfully deleted
I20260925 00:48:23.178319 2098 catalog_manager.cc:4841] TS d858e929a8c34ebdb86ce6e8c5482b8f (127.0.249.66:37607): tablet 12c8deab13684779a7e27fc31a2c067b (table test-table [id=e2330820183c46418059f5c1b2ddb2ea]) successfully deleted
I20260925 00:48:23.179220 2099 catalog_manager.cc:4841] TS 6ca18b6fd1c1478995d739fb93667bfc (127.0.249.67:45311): tablet 12c8deab13684779a7e27fc31a2c067b (table test-table [id=e2330820183c46418059f5c1b2ddb2ea]) successfully deleted
Sep 25 00:48:23 dist-test-slave-05m0 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:23.193512 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.179293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.0.249.65:41613)
0925 00:48:23.179421 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:23.179426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:23.181688 (+ 2262us) server_negotiation.cc:407] Connection header received
0925 00:48:23.181874 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:23.181877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:23.181942 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:23.181994 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:23.182979 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.183783 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.185036 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.185385 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.185510 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.189208 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:23.189219 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:23.189235 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:23.189263 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:23.191408 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:23.191892 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:23.191899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:23.191901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:23.192000 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:23.192317 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:23.192320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:23.192322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:23.192646 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:23.192842 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:23.193066 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 00:48:23.193188 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 00:48:23.193917 2310 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44154
I20260925 00:48:23.205224 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:23.195630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.0.249.126:41735)
0925 00:48:23.195750 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:23.195754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:23.196150 (+ 396us) server_negotiation.cc:407] Connection header received
0925 00:48:23.196245 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:23.196248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:23.196299 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:23.196392 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:23.197234 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.198075 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.199217 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:23.199586 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:23.199689 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:23.201285 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:23.201289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:23.201292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:23.201321 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:23.203124 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:23.203584 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:23.203591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:23.203594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:23.203696 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:23.204105 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:23.204109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:23.204110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:23.204334 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:23.204482 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:23.204817 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 00:48:23.204946 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 00:48:23.205818 2112 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59302
I20260925 00:48:23.207266 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 2184
I20260925 00:48:23.216017 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 2325
I20260925 00:48:23.222633 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 2465
I20260925 00:48:23.229085 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 2080
2026-09-25T00:48:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6034 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:23 dist-test-slave-05m0 krb5kdc[2688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:23 dist-test-slave-05m0 krb5kdc[2688](info): set up 2 sockets
Sep 25 00:48:23 dist-test-slave-05m0 krb5kdc[2688](info): commencing operation
krb5kdc: starting...
W20260925 00:48:24.277266 997 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:24 dist-test-slave-05m0 krb5kdc[2688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:24.400712 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:43487
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:43487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:24.516445 2701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:24.516753 2701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:24.516803 2701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:24.520983 2701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:24.521042 2701 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 00:48:24.521064 2701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:24.521085 2701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:24.521104 2701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:24.521122 2701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:24.526242 2701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:43487
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:43487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:24.527190 2701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:24.528133 2701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:24.536684 2709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:24.536760 2701 server_base.cc:1034] running on GCE node
W20260925 00:48:24.536691 2707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:24.536690 2706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:24.537483 2701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:24.538587 2701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:24.539800 2701 hybrid_clock.cc:648] HybridClock initialized: now 1790297304539776 us; error 45 us; skew 500 ppm
Sep 25 00:48:24 dist-test-slave-05m0 krb5kdc[2688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:24.542788 2701 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:24.543768 2701 webserver.cc:460] Webserver started at http://127.0.249.126:40331/ using document root <none> and password file <none>
I20260925 00:48:24.544322 2701 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:24.544413 2701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:24.544658 2701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:24.546767 2701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "999c7ff691524eb8899b1ee5cb7529e2"
format_stamp: "Formatted at 2026-09-25 00:48:24 on dist-test-slave-05m0"
server_key: "af98ad8d365168e578d716d6dd6d499c"
server_key_iv: "497195edb969c97a987e07fcb8abb8dd"
server_key_version: "encryptionkey@0"
I20260925 00:48:24.547366 2701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "999c7ff691524eb8899b1ee5cb7529e2"
format_stamp: "Formatted at 2026-09-25 00:48:24 on dist-test-slave-05m0"
server_key: "af98ad8d365168e578d716d6dd6d499c"
server_key_iv: "497195edb969c97a987e07fcb8abb8dd"
server_key_version: "encryptionkey@0"
I20260925 00:48:24.551155 2701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 00:48:24.553531 2716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:24.554766 2701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:24.554946 2701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "999c7ff691524eb8899b1ee5cb7529e2"
format_stamp: "Formatted at 2026-09-25 00:48:24 on dist-test-slave-05m0"
server_key: "af98ad8d365168e578d716d6dd6d499c"
server_key_iv: "497195edb969c97a987e07fcb8abb8dd"
server_key_version: "encryptionkey@0"
I20260925 00:48:24.555063 2701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:24.578810 2701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:24.582190 2701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:24.582399 2701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:24.590322 2701 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:43487
I20260925 00:48:24.590349 2778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:43487 every 8 connection(s)
I20260925 00:48:24.591276 2701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:24.594044 2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:24.596755 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 2701
I20260925 00:48:24.596882 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:24.597167 997 external_mini_cluster.cc:1428] Setting key 85b287a71c7b42cf52fd3cfcf74763b6
I20260925 00:48:24.600070 2779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2: Bootstrap starting.
I20260925 00:48:24.602264 2779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:24.603181 2779 log.cc:824] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:24.605455 2779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2: No bootstrap required, opened a new log
Sep 25 00:48:24 dist-test-slave-05m0 krb5kdc[2688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:24.610249 2779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999c7ff691524eb8899b1ee5cb7529e2" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43487 } }
I20260925 00:48:24.610530 2779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:24.610595 2779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999c7ff691524eb8899b1ee5cb7529e2, State: Initialized, Role: FOLLOWER
I20260925 00:48:24.611104 2779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999c7ff691524eb8899b1ee5cb7529e2" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43487 } }
I20260925 00:48:24.611261 2779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:24.611325 2779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:24.611423 2779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:24.612421 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:24.598619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46954 (local address 127.0.249.126:43487)
0925 00:48:24.599060 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:24.599075 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:24.599109 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:48:24.599710 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:24.599726 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:24.600060 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:24.600428 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:24.601233 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.602318 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.603609 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.604260 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.604389 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:24.607203 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:24.607209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:24.607226 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:24.607258 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:24.609210 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:24.609705 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:24.609710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:24.609716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:24.609852 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:24.610118 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:24.610121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:24.610123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:24.610641 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:24.610841 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:24.611162 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 00:48:24.611471 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":129,"threads_started":1}
I20260925 00:48:24.612486 2779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999c7ff691524eb8899b1ee5cb7529e2" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43487 } }
I20260925 00:48:24.612807 2779 leader_election.cc:302] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999c7ff691524eb8899b1ee5cb7529e2; no voters:
I20260925 00:48:24.613137 2779 leader_election.cc:288] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:24.613255 2784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:24.613487 2784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [term 1 LEADER]: Becoming Leader. State: Replica: 999c7ff691524eb8899b1ee5cb7529e2, State: Running, Role: LEADER
I20260925 00:48:24.613798 2784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999c7ff691524eb8899b1ee5cb7529e2" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43487 } }
I20260925 00:48:24.615330 2779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:24.615331 2786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "999c7ff691524eb8899b1ee5cb7529e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999c7ff691524eb8899b1ee5cb7529e2" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43487 } } }
I20260925 00:48:24.615509 2786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:24.615460 2785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999c7ff691524eb8899b1ee5cb7529e2. Latest consensus state: current_term: 1 leader_uuid: "999c7ff691524eb8899b1ee5cb7529e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999c7ff691524eb8899b1ee5cb7529e2" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43487 } } }
I20260925 00:48:24.615614 2785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:24.616767 2788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:24.619697 2788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:24.625494 2788 catalog_manager.cc:1347] Generated new cluster ID: 03fdba594e7c4f5593caa93b298f5413
I20260925 00:48:24.625571 2788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:24.639456 2788 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:24.640394 2788 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:24.645846 2788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2: Generated new TSK 0
I20260925 00:48:24.646628 2788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:48:24.682749 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43487
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:24.793054 2805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:24.793347 2805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:24.793457 2805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:24.797498 2805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:24.797587 2805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:24.797658 2805 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 00:48:24.797700 2805 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:48:24.797768 2805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:24.802985 2805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.249.126:43487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:24.804059 2805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:24.805085 2805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:24.814394 2810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:24.814379 2813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:24.814379 2811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:24.814664 2805 server_base.cc:1034] running on GCE node
I20260925 00:48:24.815218 2805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:24.815863 2805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:24.817016 2805 hybrid_clock.cc:648] HybridClock initialized: now 1790297304816989 us; error 43 us; skew 500 ppm
Sep 25 00:48:24 dist-test-slave-05m0 krb5kdc[2688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:24.819901 2805 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:24.820765 2805 webserver.cc:460] Webserver started at http://127.0.249.65:33441/ using document root <none> and password file <none>
I20260925 00:48:24.821321 2805 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:24.821431 2805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:24.821636 2805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:24.823585 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "c18d9365982d49178ddd5361ac461f0a"
format_stamp: "Formatted at 2026-09-25 00:48:24 on dist-test-slave-05m0"
server_key: "0f13b913a629dc5fe6944655f26124c0"
server_key_iv: "19f72153fb1e93e43ec8cf03a9a91cbf"
server_key_version: "encryptionkey@0"
I20260925 00:48:24.824163 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "c18d9365982d49178ddd5361ac461f0a"
format_stamp: "Formatted at 2026-09-25 00:48:24 on dist-test-slave-05m0"
server_key: "0f13b913a629dc5fe6944655f26124c0"
server_key_iv: "19f72153fb1e93e43ec8cf03a9a91cbf"
server_key_version: "encryptionkey@0"
I20260925 00:48:24.827760 2805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 00:48:24.830235 2820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:24.831496 2805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:24.831621 2805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "c18d9365982d49178ddd5361ac461f0a"
format_stamp: "Formatted at 2026-09-25 00:48:24 on dist-test-slave-05m0"
server_key: "0f13b913a629dc5fe6944655f26124c0"
server_key_iv: "19f72153fb1e93e43ec8cf03a9a91cbf"
server_key_version: "encryptionkey@0"
I20260925 00:48:24.831727 2805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:24.845289 2805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:24.848038 2805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:24.848233 2805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:24.849676 2805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:24.849748 2805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:24.849822 2805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:24.849864 2805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:48:24 dist-test-slave-05m0 krb5kdc[2688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:24.860870 2805 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:45507
I20260925 00:48:24.860932 2935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:45507 every 8 connection(s)
I20260925 00:48:24.861832 2805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:24.863761 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:24.850505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:35795 (local address 127.0.249.126:43487)
0925 00:48:24.850640 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:24.850644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:24.851669 (+ 1025us) server_negotiation.cc:407] Connection header received
0925 00:48:24.852671 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:24.852674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:24.852734 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:24.852827 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:24.854469 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.855091 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.856966 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.857295 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.857441 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:24.860057 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:24.860061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:24.860064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:24.860095 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:24.861663 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:24.862233 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:24.862238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:24.862239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:24.862334 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:24.862750 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:24.862753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:24.862755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:24.862960 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:24.863043 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:24.863538 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 00:48:24.863640 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:48:24.864542 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:24.851064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43487 (local address 127.0.249.65:35795)
0925 00:48:24.851514 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:24.851567 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:24.852458 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:24.853036 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:24.853045 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:24.853604 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:24.854286 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:24.854311 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.855229 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.855233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:24.856824 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:24.856839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.857483 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.857487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:24.857716 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:24.858270 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:24.858284 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:24.859887 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:24.861777 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:24.861783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:24.861796 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:24.862115 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:24.862441 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:24.862444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:24.862446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:24.862621 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:24.863063 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:24.863069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:24.863320 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 00:48:24.863499 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 00:48:24.863805 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":140,"threads_started":1}
I20260925 00:48:24.865836 2936 heartbeater.cc:347] Connected to a master server at 127.0.249.126:43487
I20260925 00:48:24.865986 2936 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:24.866433 2936 heartbeater.cc:511] Master 127.0.249.126:43487 requested a full tablet report, sending...
I20260925 00:48:24.867954 2732 ts_manager.cc:194] Registered new tserver with Master: c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507)
I20260925 00:48:24.869076 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 2805
I20260925 00:48:24.869191 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:24.869293 2732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:35795
I20260925 00:48:24.869504 997 external_mini_cluster.cc:1428] Setting key 253993398c03f675ccbe6c7fd84b0eea
I20260925 00:48:24.875605 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:24.867985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:35497 (local address 127.0.249.126:43487)
0925 00:48:24.868121 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:24.868126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:24.868257 (+ 131us) server_negotiation.cc:407] Connection header received
0925 00:48:24.868364 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:24.868366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:24.868407 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:24.868470 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:24.868955 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.869695 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.871040 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.871260 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.871368 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:24.872469 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:24.872474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:24.872479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:24.872507 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:24.873938 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:24.874383 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:24.874387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:24.874389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:24.874465 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:24.874790 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:24.874794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:24.874795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:24.874959 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:24.875087 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:24.875345 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 00:48:24.875452 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:48:24.875625 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:24.867926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43487 (local address 127.0.249.65:35497)
0925 00:48:24.868174 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:24.868189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:24.868284 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:24.868509 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:24.868512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:24.868708 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:24.868814 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:24.868821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.869938 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.869940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:24.870893 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:24.870904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:24.871369 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:24.871373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:24.871469 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:24.871961 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:24.871974 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:24.872307 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:24.874068 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:24.874074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:24.874077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:24.874282 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:24.874570 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:24.874573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:24.874597 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:24.874677 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:24.875090 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:24.875094 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:24.875186 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 00:48:24.875291 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 00:48:24.875427 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:24.896751 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43487
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:25.006888 2945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:25.007133 2945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:25.007180 2945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:25.011196 2945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:25.011250 2945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:25.011291 2945 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 00:48:25.011340 2945 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:48:25.011410 2945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:25.016652 2945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.249.126:43487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:25.017714 2945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:25.018694 2945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:25.027844 2950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:25.027874 2951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:25.027860 2953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:25.028061 2945 server_base.cc:1034] running on GCE node
I20260925 00:48:25.028514 2945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:25.029096 2945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:25.030253 2945 hybrid_clock.cc:648] HybridClock initialized: now 1790297305030235 us; error 27 us; skew 500 ppm
Sep 25 00:48:25 dist-test-slave-05m0 krb5kdc[2688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:25.033056 2945 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:25.033962 2945 webserver.cc:460] Webserver started at http://127.0.249.66:41957/ using document root <none> and password file <none>
I20260925 00:48:25.034507 2945 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:25.034591 2945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:25.034802 2945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:25.036780 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "17c89777e0f04eb2835189502d22523d"
format_stamp: "Formatted at 2026-09-25 00:48:25 on dist-test-slave-05m0"
server_key: "edd608aceaf5cb945ec3076e3d38c390"
server_key_iv: "547673092a8864f01e3c47be7c1cc2e9"
server_key_version: "encryptionkey@0"
I20260925 00:48:25.037318 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "17c89777e0f04eb2835189502d22523d"
format_stamp: "Formatted at 2026-09-25 00:48:25 on dist-test-slave-05m0"
server_key: "edd608aceaf5cb945ec3076e3d38c390"
server_key_iv: "547673092a8864f01e3c47be7c1cc2e9"
server_key_version: "encryptionkey@0"
I20260925 00:48:25.040814 2945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:48:25.043255 2960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:25.044428 2945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:48:25.044566 2945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "17c89777e0f04eb2835189502d22523d"
format_stamp: "Formatted at 2026-09-25 00:48:25 on dist-test-slave-05m0"
server_key: "edd608aceaf5cb945ec3076e3d38c390"
server_key_iv: "547673092a8864f01e3c47be7c1cc2e9"
server_key_version: "encryptionkey@0"
I20260925 00:48:25.044687 2945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:25.070768 2945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:25.074128 2945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:25.074331 2945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:25.075692 2945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:25.075765 2945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:25.075836 2945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:25.075881 2945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:48:25 dist-test-slave-05m0 krb5kdc[2688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:25.087001 2945 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:39843
I20260925 00:48:25.087049 3075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:39843 every 8 connection(s)
I20260925 00:48:25.088042 2945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:25.089608 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.076632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:37487 (local address 127.0.249.126:43487)
0925 00:48:25.076757 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.076762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.077793 (+ 1031us) server_negotiation.cc:407] Connection header received
0925 00:48:25.078765 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.078769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.078840 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.078956 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:25.080117 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.080813 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.082522 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.082821 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.082940 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.085855 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:25.085859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:25.085861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:25.085885 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:25.087486 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.088059 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.088062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.088063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.088170 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.088546 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.088549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.088551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.088799 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:25.088898 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.089309 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.089477 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:48:25.090307 2969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.077164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43487 (local address 127.0.249.66:37487)
0925 00:48:25.077652 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.077692 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.078485 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.079182 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.079191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.079521 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:25.079967 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.079979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.080936 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.080939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.082401 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.082410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.082985 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.082989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.083168 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.083747 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:25.083783 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:25.085654 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:25.087590 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.087596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.087609 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.087946 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.088239 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.088242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.088244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.088415 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.088886 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:25.088890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:25.089098 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.089237 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.089512 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":161,"threads_started":1}
I20260925 00:48:25.091435 3076 heartbeater.cc:347] Connected to a master server at 127.0.249.126:43487
I20260925 00:48:25.091578 3076 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:25.092094 3076 heartbeater.cc:511] Master 127.0.249.126:43487 requested a full tablet report, sending...
I20260925 00:48:25.092947 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 2945
I20260925 00:48:25.093063 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:25.093151 2733 ts_manager.cc:194] Registered new tserver with Master: 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843)
I20260925 00:48:25.093381 997 external_mini_cluster.cc:1428] Setting key c7fc2286c0dfe1be74e92d441712e9ba
I20260925 00:48:25.093942 2733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:37487
I20260925 00:48:25.102707 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.093791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43487 (local address 127.0.249.66:34227)
0925 00:48:25.095670 (+ 1879us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.095682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.095798 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.096040 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.096043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.096228 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:25.096314 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.096319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.097025 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.097028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.097635 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.097643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.098260 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.098263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.098336 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.098848 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:25.098868 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:25.099251 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:25.101204 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.101207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.101208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.101469 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.101742 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.101744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.101746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.101817 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.102311 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:25.102313 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:25.102381 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.102474 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.102574 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":80,"threads_started":1}
I20260925 00:48:25.102870 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.093800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:34227 (local address 127.0.249.126:43487)
0925 00:48:25.093897 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.093901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.095762 (+ 1861us) server_negotiation.cc:407] Connection header received
0925 00:48:25.095897 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.095899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.095944 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.096024 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:25.096429 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.096906 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.097743 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.098149 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.098265 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.099385 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:25.099391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:25.099396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:25.099427 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:25.101098 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.101562 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.101565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.101567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.101654 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.101905 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.101908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.101910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.102213 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:25.102323 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.102561 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.102681 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:25.120967 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43487
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:25.229341 3085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:25.229676 3085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:25.229768 3085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:25.233867 3085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:25.233942 3085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:25.234006 3085 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 00:48:25.234048 3085 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:48:25.234117 3085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:25.239521 3085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.249.126:43487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:25.240602 3085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:25.241693 3085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:25.251269 3093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:25.251261 3090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:25.251448 3085 server_base.cc:1034] running on GCE node
W20260925 00:48:25.251260 3091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:25.251982 3085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:25.252599 3085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:25.253759 3085 hybrid_clock.cc:648] HybridClock initialized: now 1790297305253740 us; error 27 us; skew 500 ppm
Sep 25 00:48:25 dist-test-slave-05m0 krb5kdc[2688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:25.256605 3085 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:25.257447 3085 webserver.cc:460] Webserver started at http://127.0.249.67:35953/ using document root <none> and password file <none>
I20260925 00:48:25.258071 3085 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:25.258147 3085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:25.258358 3085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:25.260493 3085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "c3e8e163f7094e1dbcf37c6bce262e76"
format_stamp: "Formatted at 2026-09-25 00:48:25 on dist-test-slave-05m0"
server_key: "51c9dacc8e4f7a07c0092246925597ba"
server_key_iv: "eb6fe640767c8d808b1e3276c3d0d998"
server_key_version: "encryptionkey@0"
I20260925 00:48:25.261065 3085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "c3e8e163f7094e1dbcf37c6bce262e76"
format_stamp: "Formatted at 2026-09-25 00:48:25 on dist-test-slave-05m0"
server_key: "51c9dacc8e4f7a07c0092246925597ba"
server_key_iv: "eb6fe640767c8d808b1e3276c3d0d998"
server_key_version: "encryptionkey@0"
I20260925 00:48:25.264763 3085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:48:25.267407 3100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:25.268575 3085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:25.268711 3085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "c3e8e163f7094e1dbcf37c6bce262e76"
format_stamp: "Formatted at 2026-09-25 00:48:25 on dist-test-slave-05m0"
server_key: "51c9dacc8e4f7a07c0092246925597ba"
server_key_iv: "eb6fe640767c8d808b1e3276c3d0d998"
server_key_version: "encryptionkey@0"
I20260925 00:48:25.268819 3085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:25.286788 3085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:25.289992 3085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:25.290190 3085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:25.291596 3085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:25.291679 3085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:25.291757 3085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:25.291828 3085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:48:25 dist-test-slave-05m0 krb5kdc[2688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:25.302878 3085 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:37115
I20260925 00:48:25.303165 3215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:37115 every 8 connection(s)
I20260925 00:48:25.303974 3085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:25.305716 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.292437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:55345 (local address 127.0.249.126:43487)
0925 00:48:25.292567 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.292571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.293665 (+ 1094us) server_negotiation.cc:407] Connection header received
0925 00:48:25.294771 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.294776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.294842 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.294934 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:25.296809 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.297567 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.298741 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.298984 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.299105 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.302001 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:25.302004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:25.302006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:25.302032 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:25.303608 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.304182 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.304184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.304186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.304260 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.304669 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.304675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.304678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.304907 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:25.305019 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.305488 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.305598 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:25.306485 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.292998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43487 (local address 127.0.249.67:55345)
0925 00:48:25.293522 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.293565 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.294504 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.295422 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.295429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.295873 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:25.296598 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.296616 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.297690 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.297693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.298611 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.298620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.299155 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.299159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.299334 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.299963 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:25.299980 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:25.301831 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:25.303720 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.303726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.303740 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.304084 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.304352 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.304356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.304358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.304526 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.305019 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:25.305024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:25.305237 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.305387 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.305685 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":135,"threads_started":1}
I20260925 00:48:25.306587 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 3085
I20260925 00:48:25.306710 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:25.307025 997 external_mini_cluster.cc:1428] Setting key 7be3f0e6a465502dea23086cb87fbd90
I20260925 00:48:25.307838 3216 heartbeater.cc:347] Connected to a master server at 127.0.249.126:43487
I20260925 00:48:25.308133 3216 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:25.309216 3216 heartbeater.cc:511] Master 127.0.249.126:43487 requested a full tablet report, sending...
I20260925 00:48:25.310179 2733 ts_manager.cc:194] Registered new tserver with Master: c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67:37115)
I20260925 00:48:25.310858 2733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:55345
I20260925 00:48:25.316318 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.309987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:55717 (local address 127.0.249.126:43487)
0925 00:48:25.310091 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.310095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.310248 (+ 153us) server_negotiation.cc:407] Connection header received
0925 00:48:25.310352 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.310355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.310393 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.310475 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:25.310824 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.311445 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.312226 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.312530 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.312642 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.313698 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:25.313701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:25.313702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:25.313719 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:25.314933 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.315333 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.315336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.315337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.315402 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.315589 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.315592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.315593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.315796 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:25.315870 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.316120 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.316219 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 00:48:25.316334 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.309876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43487 (local address 127.0.249.67:55717)
0925 00:48:25.310168 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.310180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.310276 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.310483 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.310485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.310641 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:25.310726 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.310730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.311565 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.311567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.312113 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.312121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.312640 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.312642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.312706 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.313200 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:25.313218 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:25.313592 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:25.315020 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.315022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.315023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.315209 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.315463 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.315465 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.315467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.315536 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.315876 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:25.315877 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:25.315990 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.316065 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.316160 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260925 00:48:25.320436 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 00:48:25 dist-test-slave-05m0 krb5kdc[2688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 00:48:25 dist-test-slave-05m0 krb5kdc[2688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:25.352313 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.343743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.0.249.126:43487)
0925 00:48:25.343933 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.343937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.343952 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:48:25.343997 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.343999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.344042 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.344122 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:25.344809 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.345502 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.346304 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.346552 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.346646 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.349153 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:25.349156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:25.349159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:25.349184 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:25.350676 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.351097 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.351100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.351102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.351195 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.351442 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.351444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.351445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.351648 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:25.351757 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.351958 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.352103 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 00:48:25.355445 2733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:25.358484 2733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:25.372227 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.367232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.0.249.66:39843)
0925 00:48:25.367749 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.367764 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.367786 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:48:25.367863 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.367869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.368065 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.368220 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:25.368531 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.369519 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.370909 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.371547 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.371660 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.371813 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.371987 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.372081 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":81,"threads_started":1}
I20260925 00:48:25.372419 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.366420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45507 (local address 127.0.0.1:46530)
0925 00:48:25.366811 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.366864 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.366987 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.367632 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.367638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.367662 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:25.367801 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.367812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.369298 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.369302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.370590 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.370601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.371892 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.371895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.371980 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.372003 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.372173 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.372252 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":54,"thread_start_us":95,"threads_started":1}
I20260925 00:48:25.372267 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.366455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:39843 (local address 127.0.0.1:55368)
0925 00:48:25.367376 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.367389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.367465 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.368325 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.368327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.368340 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:25.368421 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.368425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.369633 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.369637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.370774 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.370784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.371673 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.371676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.371780 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.371816 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.371952 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.372075 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260925 00:48:25.372843 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.367155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:37115 (local address 127.0.0.1:38634)
0925 00:48:25.367986 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.367999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.368080 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.368752 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.368755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.368773 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:25.368859 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.368865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.370158 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.370161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.371253 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.371263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.372430 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.372436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.372553 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.372575 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.372651 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.372722 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":87,"threads_started":1}
I20260925 00:48:25.373019 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.366520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.0.249.65:45507)
0925 00:48:25.367143 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.367151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.367179 (+ 28us) server_negotiation.cc:407] Connection header received
0925 00:48:25.367240 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.367244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.367470 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.367956 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:25.368109 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.369187 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.370729 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.371580 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.371698 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.371832 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.372772 (+ 940us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.372870 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":80,"threads_started":1}
I20260925 00:48:25.373322 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.367572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.0.249.67:37115)
0925 00:48:25.368339 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.368351 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.368372 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:48:25.368439 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.368444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.368615 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.368756 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:25.368969 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.370030 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.371378 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.372139 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.372233 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.372370 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.373094 (+ 724us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.373194 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":103,"threads_started":1}
I20260925 00:48:25.375735 3009 tablet_service.cc:1467] Processing CreateTablet for tablet 49ac7a1b84da4c6dad917e3f13a03c28 (DEFAULT_TABLE table=test-table [id=bceec1d0e7724cad8f51b0a18fd26496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:25.375734 2870 tablet_service.cc:1467] Processing CreateTablet for tablet 49ac7a1b84da4c6dad917e3f13a03c28 (DEFAULT_TABLE table=test-table [id=bceec1d0e7724cad8f51b0a18fd26496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:25.376179 3150 tablet_service.cc:1467] Processing CreateTablet for tablet 49ac7a1b84da4c6dad917e3f13a03c28 (DEFAULT_TABLE table=test-table [id=bceec1d0e7724cad8f51b0a18fd26496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:25.376837 3009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ac7a1b84da4c6dad917e3f13a03c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:25.376837 2870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ac7a1b84da4c6dad917e3f13a03c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:25.376920 3150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ac7a1b84da4c6dad917e3f13a03c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:25.383339 3239 tablet_bootstrap.cc:492] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: Bootstrap starting.
I20260925 00:48:25.384719 3240 tablet_bootstrap.cc:492] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: Bootstrap starting.
I20260925 00:48:25.384819 3241 tablet_bootstrap.cc:492] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: Bootstrap starting.
I20260925 00:48:25.385536 3239 tablet_bootstrap.cc:654] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:25.386245 3239 log.cc:824] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:25.386838 3240 tablet_bootstrap.cc:654] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:25.386838 3241 tablet_bootstrap.cc:654] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:25.387652 3241 log.cc:824] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:25.387652 3240 log.cc:824] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:25.388094 3239 tablet_bootstrap.cc:492] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: No bootstrap required, opened a new log
I20260925 00:48:25.388279 3239 ts_tablet_manager.cc:1397] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 00:48:25.389532 3241 tablet_bootstrap.cc:492] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: No bootstrap required, opened a new log
I20260925 00:48:25.389532 3240 tablet_bootstrap.cc:492] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: No bootstrap required, opened a new log
I20260925 00:48:25.389757 3241 ts_tablet_manager.cc:1397] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:48:25.389757 3240 ts_tablet_manager.cc:1397] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:48:25.391513 3239 raft_consensus.cc:348] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.391752 3239 raft_consensus.cc:374] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:25.391805 3239 raft_consensus.cc:729] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e8e163f7094e1dbcf37c6bce262e76, State: Initialized, Role: FOLLOWER
I20260925 00:48:25.392302 3239 consensus_queue.cc:260] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.392733 3246 raft_consensus.cc:482] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:25.392912 3246 raft_consensus.cc:504] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.393319 3239 ts_tablet_manager.cc:1428] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 00:48:25.393680 3216 heartbeater.cc:503] Master 127.0.249.126:43487 was elected leader, sending a full tablet report...
I20260925 00:48:25.393771 3241 raft_consensus.cc:348] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.393867 3240 raft_consensus.cc:348] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.394032 3241 raft_consensus.cc:374] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:25.394055 3240 raft_consensus.cc:374] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:25.394099 3240 raft_consensus.cc:729] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18d9365982d49178ddd5361ac461f0a, State: Initialized, Role: FOLLOWER
I20260925 00:48:25.394093 3241 raft_consensus.cc:729] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c89777e0f04eb2835189502d22523d, State: Initialized, Role: FOLLOWER
I20260925 00:48:25.394557 3240 consensus_queue.cc:260] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.394750 3241 consensus_queue.cc:260] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.395529 3240 ts_tablet_manager.cc:1428] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:48:25.395556 3241 ts_tablet_manager.cc:1428] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:48:25.395816 3246 leader_election.cc:288] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507), 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843)
I20260925 00:48:25.395859 2936 heartbeater.cc:503] Master 127.0.249.126:43487 was elected leader, sending a full tablet report...
I20260925 00:48:25.396080 3076 heartbeater.cc:503] Master 127.0.249.126:43487 was elected leader, sending a full tablet report...
I20260925 00:48:25.401479 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.397167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:46187 (local address 127.0.249.65:45507)
0925 00:48:25.397371 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.397375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.397387 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:25.397454 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.397457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.397501 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.397580 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:25.397996 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.398721 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.400332 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.400880 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.400978 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.401014 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.401267 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.401317 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260925 00:48:25.401474 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.395338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45507 (local address 127.0.249.67:46187)
0925 00:48:25.395498 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.395513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.395595 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.397715 (+ 2120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.397719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.397738 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:25.397855 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.397864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.398813 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.398816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.400213 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.400223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.401000 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.401004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.401150 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.401174 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.401249 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.401312 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":68}
I20260925 00:48:25.402249 2890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ac7a1b84da4c6dad917e3f13a03c28" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18d9365982d49178ddd5361ac461f0a" is_pre_election: true
I20260925 00:48:25.402393 3249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.396826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:39843 (local address 127.0.249.67:59289)
0925 00:48:25.397981 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.397996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.398107 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.398739 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.398742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.398756 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:25.398988 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.398996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.400336 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.400339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.401371 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.401379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.402081 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.402084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.402150 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.402167 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.402226 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.402270 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":86,"thread_start_us":103,"threads_started":1}
I20260925 00:48:25.402447 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.397140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:59289 (local address 127.0.249.66:39843)
0925 00:48:25.397281 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.397287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.398473 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 00:48:25.398523 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.398528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.398576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.398642 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:25.399476 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.400202 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.401493 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.401924 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.402062 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.402105 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.402264 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.402336 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 00:48:25.402549 2890 raft_consensus.cc:2454] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 0.
I20260925 00:48:25.403018 3104 leader_election.cc:302] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18d9365982d49178ddd5361ac461f0a, c3e8e163f7094e1dbcf37c6bce262e76; no voters:
I20260925 00:48:25.403059 3026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ac7a1b84da4c6dad917e3f13a03c28" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c89777e0f04eb2835189502d22523d" is_pre_election: true
I20260925 00:48:25.403267 3246 raft_consensus.cc:2781] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:25.403326 3026 raft_consensus.cc:2454] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 0.
I20260925 00:48:25.403365 3246 raft_consensus.cc:482] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:25.403417 3246 raft_consensus.cc:3037] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:25.404484 3246 raft_consensus.cc:504] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.404805 3246 leader_election.cc:288] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 election: Requested vote from peers c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507), 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843)
I20260925 00:48:25.405094 2890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ac7a1b84da4c6dad917e3f13a03c28" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18d9365982d49178ddd5361ac461f0a"
I20260925 00:48:25.405131 3026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ac7a1b84da4c6dad917e3f13a03c28" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c89777e0f04eb2835189502d22523d"
I20260925 00:48:25.405215 2890 raft_consensus.cc:3037] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:25.405272 3026 raft_consensus.cc:3037] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:25.406399 2890 raft_consensus.cc:2454] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 1.
I20260925 00:48:25.406731 3104 leader_election.cc:302] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18d9365982d49178ddd5361ac461f0a, c3e8e163f7094e1dbcf37c6bce262e76; no voters:
I20260925 00:48:25.406832 3026 raft_consensus.cc:2454] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 1.
I20260925 00:48:25.406906 3246 raft_consensus.cc:2781] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:25.407008 3246 raft_consensus.cc:686] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 LEADER]: Becoming Leader. State: Replica: c3e8e163f7094e1dbcf37c6bce262e76, State: Running, Role: LEADER
I20260925 00:48:25.407308 3246 consensus_queue.cc:237] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.410754 2732 catalog_manager.cc:5495] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67). New cstate: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } health_report { overall_health: HEALTHY } } }
Sep 25 00:48:25 dist-test-slave-05m0 krb5kdc[2688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:25.451759 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.443353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.0.249.126:43487)
0925 00:48:25.443479 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.443483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.443752 (+ 269us) server_negotiation.cc:407] Connection header received
0925 00:48:25.443869 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.443871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.443911 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.444005 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:25.444655 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.445241 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.446052 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.446332 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.446416 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.448720 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:25.448723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:25.448725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:25.448744 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:25.450036 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.450519 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.450523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.450524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.450598 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:25.450862 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:25.450865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:25.450866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:25.451052 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:25.451152 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.451446 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.451608 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":46}
I20260925 00:48:25.451761 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.443329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43487 (local address 127.0.0.1:46968)
0925 00:48:25.443657 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.443674 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.443786 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.444046 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.444048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.444431 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:25.444523 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.444529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.445336 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.445338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.445954 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.445962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.446436 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.446438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.446503 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.447097 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:25.447108 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:25.448549 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:25.450166 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.450175 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.450181 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.450402 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.450690 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:25.450692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:25.450694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:25.450770 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:25.451153 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:25.451161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:25.451273 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.451390 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.451551 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":147,"threads_started":1}
I20260925 00:48:25.455699 2732 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:46968:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 00:48:25.456466 2732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:25.461671 3150 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac448e28e18473c8b8f6efc94ad8c59 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94f516d8cf9546598ff455cf0ec912e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:48:25.461525 2870 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac448e28e18473c8b8f6efc94ad8c59 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94f516d8cf9546598ff455cf0ec912e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:48:25.461694 3009 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac448e28e18473c8b8f6efc94ad8c59 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94f516d8cf9546598ff455cf0ec912e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:48:25.462296 3150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac448e28e18473c8b8f6efc94ad8c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:25.462296 2870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac448e28e18473c8b8f6efc94ad8c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:25.462296 3009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac448e28e18473c8b8f6efc94ad8c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:25.464789 3239 tablet_bootstrap.cc:492] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: Bootstrap starting.
I20260925 00:48:25.465163 3240 tablet_bootstrap.cc:492] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: Bootstrap starting.
I20260925 00:48:25.465812 3239 tablet_bootstrap.cc:654] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:25.466141 3240 tablet_bootstrap.cc:654] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:25.466794 3241 tablet_bootstrap.cc:492] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: Bootstrap starting.
I20260925 00:48:25.467329 3240 tablet_bootstrap.cc:492] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: No bootstrap required, opened a new log
I20260925 00:48:25.467439 3240 ts_tablet_manager.cc:1397] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:25.467644 3241 tablet_bootstrap.cc:654] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:25.467926 3240 raft_consensus.cc:348] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.468034 3240 raft_consensus.cc:374] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:25.468075 3240 raft_consensus.cc:729] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18d9365982d49178ddd5361ac461f0a, State: Initialized, Role: FOLLOWER
I20260925 00:48:25.468189 3240 consensus_queue.cc:260] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.468502 3240 ts_tablet_manager.cc:1428] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:48:25.468448 3247 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.468604 3247 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:25.468852 3241 tablet_bootstrap.cc:492] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: No bootstrap required, opened a new log
I20260925 00:48:25.468907 3241 ts_tablet_manager.cc:1397] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:25.469105 3239 tablet_bootstrap.cc:492] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: No bootstrap required, opened a new log
I20260925 00:48:25.469188 3239 ts_tablet_manager.cc:1397] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 00:48:25.469338 3241 raft_consensus.cc:348] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.469491 3241 raft_consensus.cc:374] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:25.469532 3241 raft_consensus.cc:729] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c89777e0f04eb2835189502d22523d, State: Initialized, Role: FOLLOWER
I20260925 00:48:25.469653 3241 consensus_queue.cc:260] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.469657 3239 raft_consensus.cc:348] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.469790 3239 raft_consensus.cc:374] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:25.469835 3239 raft_consensus.cc:729] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e8e163f7094e1dbcf37c6bce262e76, State: Initialized, Role: FOLLOWER
I20260925 00:48:25.469934 3241 ts_tablet_manager.cc:1428] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:25.469974 3239 consensus_queue.cc:260] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.470155 3248 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.470268 3239 ts_tablet_manager.cc:1428] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:25.470222 3246 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.470291 3248 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:25.470336 3246 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 00:48:25.554982 3217 tablet.cc:2369] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:25.579593 3246 raft_consensus.cc:482] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:25.579737 3246 raft_consensus.cc:504] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.580240 3246 leader_election.cc:288] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507), 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843)
I20260925 00:48:25.580827 2890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac448e28e18473c8b8f6efc94ad8c59" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18d9365982d49178ddd5361ac461f0a" is_pre_election: true
I20260925 00:48:25.580873 3026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac448e28e18473c8b8f6efc94ad8c59" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c89777e0f04eb2835189502d22523d" is_pre_election: true
I20260925 00:48:25.580996 2890 raft_consensus.cc:2454] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 0.
I20260925 00:48:25.581080 3026 raft_consensus.cc:2454] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 0.
I20260925 00:48:25.581475 3104 leader_election.cc:302] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18d9365982d49178ddd5361ac461f0a, c3e8e163f7094e1dbcf37c6bce262e76; no voters:
I20260925 00:48:25.581730 3246 raft_consensus.cc:2781] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:25.581815 3246 raft_consensus.cc:482] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:25.581867 3246 raft_consensus.cc:3037] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:25.582864 3246 raft_consensus.cc:504] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.583211 3246 leader_election.cc:288] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 election: Requested vote from peers c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507), 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843)
I20260925 00:48:25.583664 2890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac448e28e18473c8b8f6efc94ad8c59" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18d9365982d49178ddd5361ac461f0a"
I20260925 00:48:25.583662 3026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac448e28e18473c8b8f6efc94ad8c59" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c89777e0f04eb2835189502d22523d"
I20260925 00:48:25.583792 3026 raft_consensus.cc:3037] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:25.583792 2890 raft_consensus.cc:3037] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:25.584962 3026 raft_consensus.cc:2454] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 1.
I20260925 00:48:25.584962 2890 raft_consensus.cc:2454] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 1.
I20260925 00:48:25.585284 3104 leader_election.cc:302] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18d9365982d49178ddd5361ac461f0a, c3e8e163f7094e1dbcf37c6bce262e76; no voters:
I20260925 00:48:25.585500 3246 raft_consensus.cc:2781] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:25.585775 3246 raft_consensus.cc:686] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 LEADER]: Becoming Leader. State: Replica: c3e8e163f7094e1dbcf37c6bce262e76, State: Running, Role: LEADER
I20260925 00:48:25.585927 3246 consensus_queue.cc:237] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.587024 3245 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: TxnStatusTablet state changed. Reason: New leader c3e8e163f7094e1dbcf37c6bce262e76. Latest consensus state: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.587165 3245 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:48:25.587554 3261 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 00:48:25.587798 3261 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 00:48:25.587850 2732 catalog_manager.cc:5495] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67). New cstate: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } health_report { overall_health: HEALTHY } } }
W20260925 00:48:25.589182 3077 tablet.cc:2369] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:48:25.613150 2937 tablet.cc:2369] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:25.626507 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.622290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.0.249.67:37115)
0925 00:48:25.622423 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.622427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.622555 (+ 128us) server_negotiation.cc:407] Connection header received
0925 00:48:25.622753 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.622755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.622808 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.622891 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:25.623156 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.623643 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.624914 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.625219 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.625303 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.625697 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.626124 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.626208 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.626342 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.626396 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:48:25.626581 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.622193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:37115 (local address 127.0.0.1:38640)
0925 00:48:25.622464 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.622481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.622617 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.622932 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.622936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.622944 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:25.623062 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.623069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.623749 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.623753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.624755 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.624768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.625338 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.625343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.625476 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.625536 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.626225 (+ 689us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.626241 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.626326 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.626421 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260925 00:48:25.632149 2890 raft_consensus.cc:1261] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Refusing update from remote peer c3e8e163f7094e1dbcf37c6bce262e76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:25.632149 3026 raft_consensus.cc:1261] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Refusing update from remote peer c3e8e163f7094e1dbcf37c6bce262e76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:25.632912 3245 consensus_queue.cc:1035] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:25.633137 3246 consensus_queue.cc:1035] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:25.637483 3248 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: TxnStatusTablet state changed. Reason: New leader c3e8e163f7094e1dbcf37c6bce262e76. Latest consensus state: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.637619 3248 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:25.637723 3247 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: TxnStatusTablet state changed. Reason: New leader c3e8e163f7094e1dbcf37c6bce262e76. Latest consensus state: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.637892 3247 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:25.641587 3245 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.641731 3245 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:48:25.642470 3246 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.642565 3246 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P c3e8e163f7094e1dbcf37c6bce262e76: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:48:25.643545 3247 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.643666 3247 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P c18d9365982d49178ddd5361ac461f0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:25.643906 3248 tablet_replica.cc:439] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } } }
I20260925 00:48:25.644013 3248 tablet_replica.cc:442] T 2ac448e28e18473c8b8f6efc94ad8c59 P 17c89777e0f04eb2835189502d22523d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:25.644711 3266 mvcc.cc:205] Tried to move back new op lower bound from 7333057763862147072 to 7333057763681501184. Current Snapshot: MvccSnapshot[applied={T|T < 7333057763862147072}]
I20260925 00:48:25.647109 3264 mvcc.cc:205] Tried to move back new op lower bound from 7333057763862147072 to 7333057763681501184. Current Snapshot: MvccSnapshot[applied={T|T < 7333057763862147072 or (T in {7333057763862147072})}]
I20260925 00:48:25.647519 3265 mvcc.cc:205] Tried to move back new op lower bound from 7333057763862147072 to 7333057763681501184. Current Snapshot: MvccSnapshot[applied={T|T < 7333057763862147072}]
I20260925 00:48:25.653211 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.650043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.0.249.67:37115)
0925 00:48:25.650149 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.650153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.650263 (+ 110us) server_negotiation.cc:407] Connection header received
0925 00:48:25.650424 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.650427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.650473 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.650584 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:25.650832 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.651333 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.652194 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.652419 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.652521 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.652781 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.652858 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.652936 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.653024 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.653077 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:48:25.663951 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.660505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:37115 (local address 127.0.249.67:48665)
0925 00:48:25.660645 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.660658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.660735 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.660995 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.660997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.661002 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:25.661132 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.661136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.662115 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.662117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.662715 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.662723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.663284 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.663286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.663350 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.663390 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.663704 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.663708 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.663753 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.663811 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":20}
I20260925 00:48:25.663992 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.660572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:48665 (local address 127.0.249.67:37115)
0925 00:48:25.660689 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.660694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.660711 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:48:25.660838 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.660841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.660894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.660974 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:25.661224 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.662011 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.662835 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.663177 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.663292 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.663532 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.663612 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.663687 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.663769 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.663836 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":50}
I20260925 00:48:25.670955 3026 raft_consensus.cc:1261] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Refusing update from remote peer c3e8e163f7094e1dbcf37c6bce262e76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:25.671000 2890 raft_consensus.cc:1261] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Refusing update from remote peer c3e8e163f7094e1dbcf37c6bce262e76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:25.671363 3245 consensus_queue.cc:1035] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:25.671487 3246 consensus_queue.cc:1035] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:25.711303 2733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:25.711462 2733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:25.713740 2733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2: Sending DeleteTablet for 3 replicas of tablet 49ac7a1b84da4c6dad917e3f13a03c28
I20260925 00:48:25.714447 3149 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ac7a1b84da4c6dad917e3f13a03c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:25 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:38634
I20260925 00:48:25.714434 3009 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ac7a1b84da4c6dad917e3f13a03c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:25 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:55368
I20260925 00:48:25.714622 2870 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ac7a1b84da4c6dad917e3f13a03c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:25 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:46530
I20260925 00:48:25.714896 3280 tablet_replica.cc:330] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: stopping tablet replica
I20260925 00:48:25.715143 3281 tablet_replica.cc:330] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: stopping tablet replica
I20260925 00:48:25.715217 3279 tablet_replica.cc:330] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: stopping tablet replica
I20260925 00:48:25.715428 3281 raft_consensus.cc:2229] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:25.715428 3279 raft_consensus.cc:2229] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:25.715615 3281 raft_consensus.cc:2258] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:25.715615 3279 raft_consensus.cc:2258] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:25.715991 3280 raft_consensus.cc:2229] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:25.716161 2733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 00:48:25.716430 3280 raft_consensus.cc:2258] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 00:48:25.716614 2733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:25.717149 3281 ts_tablet_manager.cc:1905] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:25.718508 3280 ts_tablet_manager.cc:1905] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:25.719077 3279 ts_tablet_manager.cc:1905] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:25.720293 3281 ts_tablet_manager.cc:1918] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:25.720422 3281 log.cc:1196] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/49ac7a1b84da4c6dad917e3f13a03c28
I20260925 00:48:25.720727 3281 ts_tablet_manager.cc:1939] T 49ac7a1b84da4c6dad917e3f13a03c28 P c18d9365982d49178ddd5361ac461f0a: Deleting consensus metadata
I20260925 00:48:25.722596 3280 ts_tablet_manager.cc:1918] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:25.722687 3280 log.cc:1196] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/49ac7a1b84da4c6dad917e3f13a03c28
I20260925 00:48:25.722815 2720 catalog_manager.cc:4841] TS c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507): tablet 49ac7a1b84da4c6dad917e3f13a03c28 (table test-table [id=bceec1d0e7724cad8f51b0a18fd26496]) successfully deleted
I20260925 00:48:25.723001 3280 ts_tablet_manager.cc:1939] T 49ac7a1b84da4c6dad917e3f13a03c28 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting consensus metadata
I20260925 00:48:25.723088 3149 tablet_service.cc:1467] Processing CreateTablet for tablet 70516e97c2e2430f9be70063a79651c9 (DEFAULT_TABLE table=test-table [id=b7d8ab19b4e14103804f00746719ecdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:25.723374 3149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70516e97c2e2430f9be70063a79651c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:25.723570 2870 tablet_service.cc:1467] Processing CreateTablet for tablet 70516e97c2e2430f9be70063a79651c9 (DEFAULT_TABLE table=test-table [id=b7d8ab19b4e14103804f00746719ecdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:25.723855 2870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70516e97c2e2430f9be70063a79651c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:25.725164 3239 tablet_bootstrap.cc:492] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: Bootstrap starting.
I20260925 00:48:25.725872 3009 tablet_service.cc:1467] Processing CreateTablet for tablet 70516e97c2e2430f9be70063a79651c9 (DEFAULT_TABLE table=test-table [id=b7d8ab19b4e14103804f00746719ecdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:25.726119 3009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70516e97c2e2430f9be70063a79651c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:25.726320 3240 tablet_bootstrap.cc:492] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: Bootstrap starting.
I20260925 00:48:25.727272 3279 ts_tablet_manager.cc:1918] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:25.727353 3279 log.cc:1196] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/49ac7a1b84da4c6dad917e3f13a03c28
I20260925 00:48:25.727375 3240 tablet_bootstrap.cc:654] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:25.727661 3279 ts_tablet_manager.cc:1939] T 49ac7a1b84da4c6dad917e3f13a03c28 P 17c89777e0f04eb2835189502d22523d: Deleting consensus metadata
I20260925 00:48:25.728001 3239 tablet_bootstrap.cc:654] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:25.728916 3240 tablet_bootstrap.cc:492] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: No bootstrap required, opened a new log
I20260925 00:48:25.729008 3240 ts_tablet_manager.cc:1397] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 00:48:25.729490 3240 raft_consensus.cc:348] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.729610 3240 raft_consensus.cc:374] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:25.729647 3240 raft_consensus.cc:729] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18d9365982d49178ddd5361ac461f0a, State: Initialized, Role: FOLLOWER
I20260925 00:48:25.729699 3241 tablet_bootstrap.cc:492] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: Bootstrap starting.
I20260925 00:48:25.729777 3240 consensus_queue.cc:260] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.729897 2719 catalog_manager.cc:4841] TS 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843): tablet 49ac7a1b84da4c6dad917e3f13a03c28 (table test-table [id=bceec1d0e7724cad8f51b0a18fd26496]) successfully deleted
I20260925 00:48:25.730077 3240 ts_tablet_manager.cc:1428] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:25.730551 3241 tablet_bootstrap.cc:654] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:25.730623 2720 catalog_manager.cc:4841] TS c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67:37115): tablet 49ac7a1b84da4c6dad917e3f13a03c28 (table test-table [id=bceec1d0e7724cad8f51b0a18fd26496]) successfully deleted
I20260925 00:48:25.731899 3241 tablet_bootstrap.cc:492] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: No bootstrap required, opened a new log
I20260925 00:48:25.731983 3241 ts_tablet_manager.cc:1397] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:25.732383 3241 raft_consensus.cc:348] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.732498 3241 raft_consensus.cc:374] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:25.732534 3241 raft_consensus.cc:729] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c89777e0f04eb2835189502d22523d, State: Initialized, Role: FOLLOWER
I20260925 00:48:25.732668 3241 consensus_queue.cc:260] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.732913 3241 ts_tablet_manager.cc:1428] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:25.732910 3239 tablet_bootstrap.cc:492] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: No bootstrap required, opened a new log
I20260925 00:48:25.733923 3239 ts_tablet_manager.cc:1397] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 00:48:25.734442 3239 raft_consensus.cc:348] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.734573 3239 raft_consensus.cc:374] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:25.734609 3239 raft_consensus.cc:729] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e8e163f7094e1dbcf37c6bce262e76, State: Initialized, Role: FOLLOWER
I20260925 00:48:25.734740 3239 consensus_queue.cc:260] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.735074 3239 ts_tablet_manager.cc:1428] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:25.820015 3248 raft_consensus.cc:482] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:25.820171 3248 raft_consensus.cc:504] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.821259 3248 leader_election.cc:288] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507), c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67:37115)
I20260925 00:48:25.825959 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.822048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:42003 (local address 127.0.249.67:37115)
0925 00:48:25.822163 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.822168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.822294 (+ 126us) server_negotiation.cc:407] Connection header received
0925 00:48:25.822409 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.822412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.822463 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.822558 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:25.822833 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.823558 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.824843 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.825347 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.825514 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.825579 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.825783 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.825850 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:48:25.825992 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.821835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:37115 (local address 127.0.249.66:42003)
0925 00:48:25.822196 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.822210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.822318 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.822585 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.822588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.822606 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:25.822711 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.822717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.823684 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.823687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.824736 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.824743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.825500 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.825503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.825607 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.825665 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.825762 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.825833 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":29,"spinlock_wait_cycles":7808,"thread_start_us":85,"threads_started":1}
I20260925 00:48:25.826134 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.821593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:34321 (local address 127.0.249.65:45507)
0925 00:48:25.821745 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.821750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.821835 (+ 85us) server_negotiation.cc:407] Connection header received
0925 00:48:25.822005 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.822008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.822064 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.822120 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:25.822544 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.823151 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.824657 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.825280 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.825448 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.825510 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.825914 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.825996 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:48:25.826551 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.821457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45507 (local address 127.0.249.66:34321)
0925 00:48:25.821728 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.821746 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.821857 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.822270 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.822274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.822313 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:25.822437 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.822442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.823294 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.823297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.824491 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.824500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.825608 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.825613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.825742 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.825768 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.826326 (+ 558us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.826405 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":1,"spinlock_wait_cycles":27776,"thread_start_us":100,"threads_started":1}
I20260925 00:48:25.826535 3170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70516e97c2e2430f9be70063a79651c9" candidate_uuid: "17c89777e0f04eb2835189502d22523d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" is_pre_election: true
I20260925 00:48:25.826802 3170 raft_consensus.cc:2454] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17c89777e0f04eb2835189502d22523d in term 0.
I20260925 00:48:25.827028 2890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70516e97c2e2430f9be70063a79651c9" candidate_uuid: "17c89777e0f04eb2835189502d22523d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18d9365982d49178ddd5361ac461f0a" is_pre_election: true
I20260925 00:48:25.827163 2890 raft_consensus.cc:2454] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17c89777e0f04eb2835189502d22523d in term 0.
I20260925 00:48:25.827227 2964 leader_election.cc:302] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c89777e0f04eb2835189502d22523d, c3e8e163f7094e1dbcf37c6bce262e76; no voters:
I20260925 00:48:25.827485 3248 raft_consensus.cc:2781] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:25.827539 3248 raft_consensus.cc:482] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:25.827577 3248 raft_consensus.cc:3037] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:25.828526 3248 raft_consensus.cc:504] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.828881 3248 leader_election.cc:288] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [CANDIDATE]: Term 1 election: Requested vote from peers c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507), c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67:37115)
I20260925 00:48:25.829190 2890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70516e97c2e2430f9be70063a79651c9" candidate_uuid: "17c89777e0f04eb2835189502d22523d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18d9365982d49178ddd5361ac461f0a"
I20260925 00:48:25.829209 3170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70516e97c2e2430f9be70063a79651c9" candidate_uuid: "17c89777e0f04eb2835189502d22523d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e8e163f7094e1dbcf37c6bce262e76"
I20260925 00:48:25.829319 2890 raft_consensus.cc:3037] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:25.829336 3170 raft_consensus.cc:3037] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:25.830235 3170 raft_consensus.cc:2454] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c89777e0f04eb2835189502d22523d in term 1.
I20260925 00:48:25.830533 2890 raft_consensus.cc:2454] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c89777e0f04eb2835189502d22523d in term 1.
I20260925 00:48:25.830684 2964 leader_election.cc:302] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c89777e0f04eb2835189502d22523d, c3e8e163f7094e1dbcf37c6bce262e76; no voters:
I20260925 00:48:25.830940 3248 raft_consensus.cc:2781] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:25.831142 3248 raft_consensus.cc:686] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 1 LEADER]: Becoming Leader. State: Replica: 17c89777e0f04eb2835189502d22523d, State: Running, Role: LEADER
I20260925 00:48:25.831390 3248 consensus_queue.cc:237] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:25.833963 2730 catalog_manager.cc:5495] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d reported cstate change: term changed from 0 to 1, leader changed from <none> to 17c89777e0f04eb2835189502d22523d (127.0.249.66). New cstate: current_term: 1 leader_uuid: "17c89777e0f04eb2835189502d22523d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } health_report { overall_health: UNKNOWN } } }
I20260925 00:48:25.873442 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.869099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.0.249.66:39843)
0925 00:48:25.869216 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.869220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.869235 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:48:25.869309 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.869312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.869368 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.869485 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:25.869776 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.870543 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.871722 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.872067 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.872179 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.872473 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.873004 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.873115 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.873230 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.873295 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:48:25.885214 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.881608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:38409 (local address 127.0.249.67:37115)
0925 00:48:25.881735 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.881738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.881912 (+ 174us) server_negotiation.cc:407] Connection header received
0925 00:48:25.882057 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.882059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.882098 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.882159 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:25.882475 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.882951 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.884145 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.884346 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.884419 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.884783 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.884845 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.884896 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.885071 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.885114 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:25.885316 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.881465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:37115 (local address 127.0.249.66:38409)
0925 00:48:25.881804 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.881834 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.881956 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.882201 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.882213 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.882223 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:25.882364 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.882372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.883070 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.883074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.884027 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.884040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.884461 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.884466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.884587 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.884646 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.884936 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.884944 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.885036 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.885122 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260925 00:48:25.891336 2890 raft_consensus.cc:1261] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Refusing update from remote peer 17c89777e0f04eb2835189502d22523d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:25.891693 3170 raft_consensus.cc:1261] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Refusing update from remote peer 17c89777e0f04eb2835189502d22523d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:25.891796 3248 consensus_queue.cc:1035] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:25.892220 3248 consensus_queue.cc:1035] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:25.911190 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.908059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:56879 (local address 127.0.249.66:39843)
0925 00:48:25.908171 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:25.908176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:25.908320 (+ 144us) server_negotiation.cc:407] Connection header received
0925 00:48:25.908355 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:25.908357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:25.908391 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:25.908471 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:25.908722 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.909397 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.910192 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.910421 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.910527 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.910754 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.910827 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.910900 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:25.911015 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 00:48:25.911067 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:48:25.911190 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:25.908007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:39843 (local address 127.0.249.67:56879)
0925 00:48:25.908159 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:25.908175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:25.908259 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:25.908481 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:25.908484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:25.908490 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:25.908606 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.908612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.909522 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.909526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.910078 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:25.910086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:25.910525 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:25.910537 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:25.910618 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:25.910642 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:25.910915 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:25.910922 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:48:25.911003 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 00:48:25.911081 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 00:48:25.912021 3266 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 00:48:27.471369 2733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:27.471549 2733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:27.473994 2733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2: Sending DeleteTablet for 3 replicas of tablet 70516e97c2e2430f9be70063a79651c9
I20260925 00:48:27.474615 3009 tablet_service.cc:1514] Processing DeleteTablet for tablet 70516e97c2e2430f9be70063a79651c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:27 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:55368
I20260925 00:48:27.474612 2870 tablet_service.cc:1514] Processing DeleteTablet for tablet 70516e97c2e2430f9be70063a79651c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:27 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:46530
I20260925 00:48:27.474661 3149 tablet_service.cc:1514] Processing DeleteTablet for tablet 70516e97c2e2430f9be70063a79651c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:27 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:38634
I20260925 00:48:27.475011 3319 tablet_replica.cc:330] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: stopping tablet replica
I20260925 00:48:27.475138 3319 raft_consensus.cc:2229] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:27.475135 3318 tablet_replica.cc:330] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: stopping tablet replica
I20260925 00:48:27.475272 3318 raft_consensus.cc:2229] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:27.475456 3319 raft_consensus.cc:2258] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:27.476183 3319 ts_tablet_manager.cc:1905] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:27.477165 3318 raft_consensus.cc:2258] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:27.477905 3318 ts_tablet_manager.cc:1905] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:27.478977 3319 ts_tablet_manager.cc:1918] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:48:27.479041 3319 log.cc:1196] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/70516e97c2e2430f9be70063a79651c9
I20260925 00:48:27.479314 3319 ts_tablet_manager.cc:1939] T 70516e97c2e2430f9be70063a79651c9 P 17c89777e0f04eb2835189502d22523d: Deleting consensus metadata
I20260925 00:48:27.479928 2719 catalog_manager.cc:4841] TS 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843): tablet 70516e97c2e2430f9be70063a79651c9 (table test-table [id=b7d8ab19b4e14103804f00746719ecdb]) successfully deleted
I20260925 00:48:27.480288 3318 ts_tablet_manager.cc:1918] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:48:27.480372 3318 log.cc:1196] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/70516e97c2e2430f9be70063a79651c9
I20260925 00:48:27.480650 3318 ts_tablet_manager.cc:1939] T 70516e97c2e2430f9be70063a79651c9 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting consensus metadata
I20260925 00:48:27.481272 3317 tablet_replica.cc:330] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: stopping tablet replica
I20260925 00:48:27.481252 2720 catalog_manager.cc:4841] TS c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67:37115): tablet 70516e97c2e2430f9be70063a79651c9 (table test-table [id=b7d8ab19b4e14103804f00746719ecdb]) successfully deleted
I20260925 00:48:27.481424 3317 raft_consensus.cc:2229] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:27.481561 3317 raft_consensus.cc:2258] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:27.482115 3317 ts_tablet_manager.cc:1905] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:27.484081 3317 ts_tablet_manager.cc:1918] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:48:27.484170 3317 log.cc:1196] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/70516e97c2e2430f9be70063a79651c9
I20260925 00:48:27.484480 3317 ts_tablet_manager.cc:1939] T 70516e97c2e2430f9be70063a79651c9 P c18d9365982d49178ddd5361ac461f0a: Deleting consensus metadata
I20260925 00:48:27.485147 2720 catalog_manager.cc:4841] TS c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507): tablet 70516e97c2e2430f9be70063a79651c9 (table test-table [id=b7d8ab19b4e14103804f00746719ecdb]) successfully deleted
Sep 25 00:48:27 dist-test-slave-05m0 krb5kdc[2688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:48:27 dist-test-slave-05m0 krb5kdc[2688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:27.506304 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.494496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.0.249.126:43487)
0925 00:48:27.494755 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:27.494762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:27.494796 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:48:27.494907 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:27.494909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:27.494953 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:27.495025 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:27.495705 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.496242 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.497022 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.497235 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.497306 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.503092 (+ 5786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:27.503096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:27.503098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:27.503123 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:27.504651 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:27.505045 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:27.505048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:27.505049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:27.505118 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:27.505336 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:27.505338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:27.505339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:27.505654 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:27.505759 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:27.505970 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 00:48:27.506077 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":111,"threads_started":1}
I20260925 00:48:27.508992 2733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:27.509375 2733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:27.514087 2870 tablet_service.cc:1467] Processing CreateTablet for tablet f64f029668b1472a8a3c00109c0a9b77 (DEFAULT_TABLE table=test-table [id=2d4a3280e0324002a8f9396ab09ad5f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:27.514163 3009 tablet_service.cc:1467] Processing CreateTablet for tablet f64f029668b1472a8a3c00109c0a9b77 (DEFAULT_TABLE table=test-table [id=2d4a3280e0324002a8f9396ab09ad5f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:27.514276 3149 tablet_service.cc:1467] Processing CreateTablet for tablet f64f029668b1472a8a3c00109c0a9b77 (DEFAULT_TABLE table=test-table [id=2d4a3280e0324002a8f9396ab09ad5f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:27.514422 2870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64f029668b1472a8a3c00109c0a9b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:27.514441 3009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64f029668b1472a8a3c00109c0a9b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:27.514513 3149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64f029668b1472a8a3c00109c0a9b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:27.517010 3329 tablet_bootstrap.cc:492] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: Bootstrap starting.
I20260925 00:48:27.517068 3330 tablet_bootstrap.cc:492] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: Bootstrap starting.
I20260925 00:48:27.517586 3328 tablet_bootstrap.cc:492] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: Bootstrap starting.
I20260925 00:48:27.517953 3329 tablet_bootstrap.cc:654] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:27.518345 3330 tablet_bootstrap.cc:654] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:27.518572 3328 tablet_bootstrap.cc:654] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:27.519160 3329 tablet_bootstrap.cc:492] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: No bootstrap required, opened a new log
I20260925 00:48:27.519248 3329 ts_tablet_manager.cc:1397] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:27.519579 3330 tablet_bootstrap.cc:492] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: No bootstrap required, opened a new log
I20260925 00:48:27.519678 3330 ts_tablet_manager.cc:1397] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 00:48:27.519665 3329 raft_consensus.cc:348] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.519797 3329 raft_consensus.cc:374] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:27.519832 3329 raft_consensus.cc:729] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e8e163f7094e1dbcf37c6bce262e76, State: Initialized, Role: FOLLOWER
I20260925 00:48:27.519990 3329 consensus_queue.cc:260] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.520180 3330 raft_consensus.cc:348] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.520301 3330 raft_consensus.cc:374] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:27.520345 3330 raft_consensus.cc:729] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18d9365982d49178ddd5361ac461f0a, State: Initialized, Role: FOLLOWER
I20260925 00:48:27.520361 3329 ts_tablet_manager.cc:1428] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:27.520468 3330 consensus_queue.cc:260] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.520856 3330 ts_tablet_manager.cc:1428] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:27.521654 3328 tablet_bootstrap.cc:492] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: No bootstrap required, opened a new log
I20260925 00:48:27.521747 3328 ts_tablet_manager.cc:1397] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 00:48:27.522168 3328 raft_consensus.cc:348] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.522279 3328 raft_consensus.cc:374] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:27.522321 3328 raft_consensus.cc:729] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c89777e0f04eb2835189502d22523d, State: Initialized, Role: FOLLOWER
I20260925 00:48:27.522439 3328 consensus_queue.cc:260] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.522724 3328 ts_tablet_manager.cc:1428] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:48:27.625092 3334 raft_consensus.cc:482] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:27.625277 3334 raft_consensus.cc:504] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.626464 3334 leader_election.cc:288] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843), c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67:37115)
I20260925 00:48:27.631660 3337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.626801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:45173 (local address 127.0.249.67:37115)
0925 00:48:27.627143 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:27.627153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:27.627549 (+ 396us) server_negotiation.cc:407] Connection header received
0925 00:48:27.627677 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:27.627680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:27.627724 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:27.628049 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:27.628283 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.628952 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.630308 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.630918 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.631048 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.631101 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:27.631441 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 00:48:27.631514 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260925 00:48:27.631660 3338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.626913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:60595 (local address 127.0.249.66:39843)
0925 00:48:27.627202 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:27.627213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:27.627263 (+ 50us) server_negotiation.cc:407] Connection header received
0925 00:48:27.627410 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:27.627412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:27.627453 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:27.627556 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:27.627843 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.628652 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.630314 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.630919 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.631049 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.631102 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:27.631441 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 00:48:27.631514 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260925 00:48:27.631734 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.626700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:37115 (local address 127.0.249.65:45173)
0925 00:48:27.627460 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:27.627470 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:27.627572 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:27.628069 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:27.628071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:27.628088 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:27.628180 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:27.628184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.629078 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.629081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:27.630202 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:27.630211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.631048 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.631052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:27.631258 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.631327 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 00:48:27.631411 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 00:48:27.631530 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":137,"thread_start_us":88,"threads_started":1}
I20260925 00:48:27.632064 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.626700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:39843 (local address 127.0.249.65:60595)
0925 00:48:27.627189 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:27.627202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:27.627311 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:27.627577 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:27.627580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:27.627597 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:27.627729 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:27.627735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.628786 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.628789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:27.630172 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:27.630183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.631048 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.631052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:27.631284 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.631330 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 00:48:27.631418 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 00:48:27.631530 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":101,"thread_start_us":130,"threads_started":1}
I20260925 00:48:27.632342 3170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64f029668b1472a8a3c00109c0a9b77" candidate_uuid: "c18d9365982d49178ddd5361ac461f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" is_pre_election: true
I20260925 00:48:27.632527 3170 raft_consensus.cc:2454] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18d9365982d49178ddd5361ac461f0a in term 0.
I20260925 00:48:27.632927 2824 leader_election.cc:302] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18d9365982d49178ddd5361ac461f0a, c3e8e163f7094e1dbcf37c6bce262e76; no voters:
I20260925 00:48:27.632961 3030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64f029668b1472a8a3c00109c0a9b77" candidate_uuid: "c18d9365982d49178ddd5361ac461f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c89777e0f04eb2835189502d22523d" is_pre_election: true
I20260925 00:48:27.633111 3030 raft_consensus.cc:2454] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18d9365982d49178ddd5361ac461f0a in term 0.
I20260925 00:48:27.633219 3334 raft_consensus.cc:2781] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:27.633272 3334 raft_consensus.cc:482] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:27.633306 3334 raft_consensus.cc:3037] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:27.634320 3334 raft_consensus.cc:504] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.634642 3334 leader_election.cc:288] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [CANDIDATE]: Term 1 election: Requested vote from peers 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843), c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67:37115)
I20260925 00:48:27.634997 3030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64f029668b1472a8a3c00109c0a9b77" candidate_uuid: "c18d9365982d49178ddd5361ac461f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c89777e0f04eb2835189502d22523d"
I20260925 00:48:27.635066 3170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64f029668b1472a8a3c00109c0a9b77" candidate_uuid: "c18d9365982d49178ddd5361ac461f0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e8e163f7094e1dbcf37c6bce262e76"
I20260925 00:48:27.635136 3030 raft_consensus.cc:3037] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:27.635190 3170 raft_consensus.cc:3037] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:27.636190 3030 raft_consensus.cc:2454] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18d9365982d49178ddd5361ac461f0a in term 1.
I20260925 00:48:27.636430 3170 raft_consensus.cc:2454] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18d9365982d49178ddd5361ac461f0a in term 1.
I20260925 00:48:27.636494 2823 leader_election.cc:302] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c89777e0f04eb2835189502d22523d, c18d9365982d49178ddd5361ac461f0a; no voters:
I20260925 00:48:27.636653 3334 raft_consensus.cc:2781] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:27.636916 3334 raft_consensus.cc:686] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 1 LEADER]: Becoming Leader. State: Replica: c18d9365982d49178ddd5361ac461f0a, State: Running, Role: LEADER
I20260925 00:48:27.637163 3334 consensus_queue.cc:237] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.639454 2730 catalog_manager.cc:5495] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a reported cstate change: term changed from 0 to 1, leader changed from <none> to c18d9365982d49178ddd5361ac461f0a (127.0.249.65). New cstate: current_term: 1 leader_uuid: "c18d9365982d49178ddd5361ac461f0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } health_report { overall_health: UNKNOWN } } }
I20260925 00:48:27.678188 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.674615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.0.249.65:45507)
0925 00:48:27.674933 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:27.674941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:27.674955 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:27.675029 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:27.675031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:27.675085 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:27.675197 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:27.675378 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.675967 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.676757 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.677058 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.677146 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.677361 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:27.677854 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:27.677950 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:27.678011 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 00:48:27.678059 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":162,"threads_started":1}
I20260925 00:48:27.687794 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.684729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:37115 (local address 127.0.249.65:48871)
0925 00:48:27.685023 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:27.685037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:27.685112 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:27.685306 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:27.685309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:27.685315 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:27.685460 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:27.685465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.686152 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.686154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:27.686764 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:27.686771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.687185 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.687187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:27.687254 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.687292 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:27.687565 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:27.687570 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:48:27.687618 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 00:48:27.687675 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":119,"threads_started":1}
I20260925 00:48:27.687793 3337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.685008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:48871 (local address 127.0.249.67:37115)
0925 00:48:27.685118 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:27.685122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:27.685135 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:48:27.685179 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:27.685181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:27.685222 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:27.685286 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:27.685550 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.686061 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.686868 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.687090 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.687170 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.687421 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:27.687482 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:27.687540 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:27.687647 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 00:48:27.687691 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 00:48:27.693472 3030 raft_consensus.cc:1261] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Refusing update from remote peer c18d9365982d49178ddd5361ac461f0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:27.693599 3170 raft_consensus.cc:1261] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Refusing update from remote peer c18d9365982d49178ddd5361ac461f0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:27.693997 3339 consensus_queue.cc:1035] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:27.694216 3334 consensus_queue.cc:1035] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:27.710678 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.707098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:43011 (local address 127.0.249.65:45507)
0925 00:48:27.707214 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:27.707224 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:27.707420 (+ 196us) server_negotiation.cc:407] Connection header received
0925 00:48:27.707587 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:27.707590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:27.707633 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:27.707702 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:27.708031 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.708633 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.709588 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.709860 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.709935 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.710183 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:27.710264 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:27.710328 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:27.710444 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 00:48:27.710525 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:48:27.710703 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.706996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45507 (local address 127.0.249.67:43011)
0925 00:48:27.707331 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:27.707343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:27.707463 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:27.707813 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:27.707818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:27.707826 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:27.707929 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:27.707933 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.708761 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.708767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:27.709445 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:27.709456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.709962 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.709965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:27.710063 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.710090 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:27.710354 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:27.710360 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:48:27.710436 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:48:27.710547 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":94,"threads_started":1}
I20260925 00:48:27.729658 2733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:27.729830 2733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:27.731813 2733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2: Sending DeleteTablet for 3 replicas of tablet f64f029668b1472a8a3c00109c0a9b77
I20260925 00:48:27.732374 2870 tablet_service.cc:1514] Processing DeleteTablet for tablet f64f029668b1472a8a3c00109c0a9b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:27 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:46530
I20260925 00:48:27.732584 3009 tablet_service.cc:1514] Processing DeleteTablet for tablet f64f029668b1472a8a3c00109c0a9b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:27 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:55368
I20260925 00:48:27.732774 3317 tablet_replica.cc:330] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: stopping tablet replica
I20260925 00:48:27.732738 3149 tablet_service.cc:1514] Processing DeleteTablet for tablet f64f029668b1472a8a3c00109c0a9b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:27 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:38634
I20260925 00:48:27.732890 3317 raft_consensus.cc:2229] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:27.733132 3317 raft_consensus.cc:2258] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:27.733304 3319 tablet_replica.cc:330] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: stopping tablet replica
I20260925 00:48:27.733506 3318 tablet_replica.cc:330] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: stopping tablet replica
I20260925 00:48:27.733604 3318 raft_consensus.cc:2229] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:27.733687 3318 raft_consensus.cc:2258] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:27.733794 3319 raft_consensus.cc:2229] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:27.733942 3319 raft_consensus.cc:2258] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:27.734759 3319 ts_tablet_manager.cc:1905] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:27.734761 2733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 00:48:27.734946 3318 ts_tablet_manager.cc:1905] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:48:27.735160 2733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:27.737057 3319 ts_tablet_manager.cc:1918] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:27.737128 3319 log.cc:1196] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/f64f029668b1472a8a3c00109c0a9b77
I20260925 00:48:27.737368 3319 ts_tablet_manager.cc:1939] T f64f029668b1472a8a3c00109c0a9b77 P 17c89777e0f04eb2835189502d22523d: Deleting consensus metadata
I20260925 00:48:27.737617 3318 ts_tablet_manager.cc:1918] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:27.737694 3318 log.cc:1196] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/f64f029668b1472a8a3c00109c0a9b77
I20260925 00:48:27.737954 3318 ts_tablet_manager.cc:1939] T f64f029668b1472a8a3c00109c0a9b77 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting consensus metadata
I20260925 00:48:27.738080 2719 catalog_manager.cc:4841] TS 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843): tablet f64f029668b1472a8a3c00109c0a9b77 (table test-table [id=2d4a3280e0324002a8f9396ab09ad5f3]) successfully deleted
I20260925 00:48:27.738130 3317 ts_tablet_manager.cc:1905] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:27.739573 2720 catalog_manager.cc:4841] TS c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67:37115): tablet f64f029668b1472a8a3c00109c0a9b77 (table test-table [id=2d4a3280e0324002a8f9396ab09ad5f3]) successfully deleted
I20260925 00:48:27.740339 3317 ts_tablet_manager.cc:1918] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:27.740391 3317 log.cc:1196] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/f64f029668b1472a8a3c00109c0a9b77
I20260925 00:48:27.740666 3317 ts_tablet_manager.cc:1939] T f64f029668b1472a8a3c00109c0a9b77 P c18d9365982d49178ddd5361ac461f0a: Deleting consensus metadata
I20260925 00:48:27.741613 2720 catalog_manager.cc:4841] TS c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507): tablet f64f029668b1472a8a3c00109c0a9b77 (table test-table [id=2d4a3280e0324002a8f9396ab09ad5f3]) successfully deleted
I20260925 00:48:27.742362 2870 tablet_service.cc:1467] Processing CreateTablet for tablet 61ffe9c9ddfd4a2c82e977e60d81d010 (DEFAULT_TABLE table=test-table [id=63e7b89c39374f94acaf165865b91063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:27.742558 3149 tablet_service.cc:1467] Processing CreateTablet for tablet 61ffe9c9ddfd4a2c82e977e60d81d010 (DEFAULT_TABLE table=test-table [id=63e7b89c39374f94acaf165865b91063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:27.742612 2870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ffe9c9ddfd4a2c82e977e60d81d010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:27.742668 3009 tablet_service.cc:1467] Processing CreateTablet for tablet 61ffe9c9ddfd4a2c82e977e60d81d010 (DEFAULT_TABLE table=test-table [id=63e7b89c39374f94acaf165865b91063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:27.742787 3149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ffe9c9ddfd4a2c82e977e60d81d010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:27.742895 3009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ffe9c9ddfd4a2c82e977e60d81d010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:27.745137 3330 tablet_bootstrap.cc:492] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a: Bootstrap starting.
I20260925 00:48:27.745527 3328 tablet_bootstrap.cc:492] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: Bootstrap starting.
I20260925 00:48:27.746037 3330 tablet_bootstrap.cc:654] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:27.746351 3328 tablet_bootstrap.cc:654] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:27.746869 3329 tablet_bootstrap.cc:492] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: Bootstrap starting.
I20260925 00:48:27.747186 3330 tablet_bootstrap.cc:492] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a: No bootstrap required, opened a new log
I20260925 00:48:27.747287 3330 ts_tablet_manager.cc:1397] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:27.747623 3329 tablet_bootstrap.cc:654] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:27.747768 3330 raft_consensus.cc:348] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.747896 3330 raft_consensus.cc:374] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:27.747934 3330 raft_consensus.cc:729] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18d9365982d49178ddd5361ac461f0a, State: Initialized, Role: FOLLOWER
I20260925 00:48:27.748064 3330 consensus_queue.cc:260] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.748355 3330 ts_tablet_manager.cc:1428] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:27.749018 3329 tablet_bootstrap.cc:492] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: No bootstrap required, opened a new log
I20260925 00:48:27.749107 3329 ts_tablet_manager.cc:1397] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:27.749550 3328 tablet_bootstrap.cc:492] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: No bootstrap required, opened a new log
I20260925 00:48:27.749643 3328 ts_tablet_manager.cc:1397] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 00:48:27.749657 3329 raft_consensus.cc:348] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.749816 3329 raft_consensus.cc:374] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:27.749862 3329 raft_consensus.cc:729] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e8e163f7094e1dbcf37c6bce262e76, State: Initialized, Role: FOLLOWER
I20260925 00:48:27.749996 3329 consensus_queue.cc:260] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.750031 3328 raft_consensus.cc:348] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.750138 3328 raft_consensus.cc:374] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:27.750172 3328 raft_consensus.cc:729] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c89777e0f04eb2835189502d22523d, State: Initialized, Role: FOLLOWER
I20260925 00:48:27.750303 3329 ts_tablet_manager.cc:1428] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:27.750285 3328 consensus_queue.cc:260] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.750526 3328 ts_tablet_manager.cc:1428] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:27.785599 3245 raft_consensus.cc:482] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:27.785756 3245 raft_consensus.cc:504] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.786196 3245 leader_election.cc:288] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507), 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843)
I20260925 00:48:27.786638 2889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ffe9c9ddfd4a2c82e977e60d81d010" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18d9365982d49178ddd5361ac461f0a" is_pre_election: true
I20260925 00:48:27.786628 3030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ffe9c9ddfd4a2c82e977e60d81d010" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c89777e0f04eb2835189502d22523d" is_pre_election: true
I20260925 00:48:27.786803 3030 raft_consensus.cc:2454] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 0.
I20260925 00:48:27.786806 2889 raft_consensus.cc:2454] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 0.
I20260925 00:48:27.787137 3103 leader_election.cc:302] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c89777e0f04eb2835189502d22523d, c3e8e163f7094e1dbcf37c6bce262e76; no voters:
I20260925 00:48:27.787369 3245 raft_consensus.cc:2781] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:27.787453 3245 raft_consensus.cc:482] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:27.787503 3245 raft_consensus.cc:3037] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:27.788501 3245 raft_consensus.cc:504] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.788846 3245 leader_election.cc:288] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 election: Requested vote from peers c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507), 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843)
I20260925 00:48:27.789165 2889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ffe9c9ddfd4a2c82e977e60d81d010" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18d9365982d49178ddd5361ac461f0a"
I20260925 00:48:27.789188 3030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ffe9c9ddfd4a2c82e977e60d81d010" candidate_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c89777e0f04eb2835189502d22523d"
I20260925 00:48:27.789300 3030 raft_consensus.cc:3037] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:27.789305 2889 raft_consensus.cc:3037] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:27.790234 2889 raft_consensus.cc:2454] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 1.
I20260925 00:48:27.790418 3030 raft_consensus.cc:2454] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e8e163f7094e1dbcf37c6bce262e76 in term 1.
I20260925 00:48:27.790514 3104 leader_election.cc:302] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18d9365982d49178ddd5361ac461f0a, c3e8e163f7094e1dbcf37c6bce262e76; no voters:
I20260925 00:48:27.790709 3245 raft_consensus.cc:2781] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:27.790798 3245 raft_consensus.cc:686] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 LEADER]: Becoming Leader. State: Replica: c3e8e163f7094e1dbcf37c6bce262e76, State: Running, Role: LEADER
I20260925 00:48:27.790970 3245 consensus_queue.cc:237] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } }
I20260925 00:48:27.792695 2733 catalog_manager.cc:5495] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67). New cstate: current_term: 1 leader_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e8e163f7094e1dbcf37c6bce262e76" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 37115 } health_report { overall_health: HEALTHY } } }
I20260925 00:48:27.810169 3337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:27.806607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.0.249.67:37115)
0925 00:48:27.806718 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:27.806725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:27.806744 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:48:27.806920 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:27.806922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:27.806973 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:27.807052 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:27.807335 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.808139 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.809068 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:27.809435 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:27.809532 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:27.809745 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:27.809825 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:27.809897 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:27.809981 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 00:48:27.810030 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:48:27.816311 2890 raft_consensus.cc:1261] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a [term 1 FOLLOWER]: Refusing update from remote peer c3e8e163f7094e1dbcf37c6bce262e76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:27.816427 3030 raft_consensus.cc:1261] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Refusing update from remote peer c3e8e163f7094e1dbcf37c6bce262e76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:27.816849 3245 consensus_queue.cc:1035] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18d9365982d49178ddd5361ac461f0a" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:27.817000 3298 consensus_queue.cc:1035] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c89777e0f04eb2835189502d22523d" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 00:48:27.831496 3296 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 00:48:29.384034 2730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:29.384181 2730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:29.386418 2730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999c7ff691524eb8899b1ee5cb7529e2: Sending DeleteTablet for 3 replicas of tablet 61ffe9c9ddfd4a2c82e977e60d81d010
I20260925 00:48:29.386999 2870 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ffe9c9ddfd4a2c82e977e60d81d010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:29 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:46530
I20260925 00:48:29.387048 3149 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ffe9c9ddfd4a2c82e977e60d81d010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:29 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:38634
I20260925 00:48:29.387100 3009 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ffe9c9ddfd4a2c82e977e60d81d010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:29 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:55368
I20260925 00:48:29.387349 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 2805
I20260925 00:48:29.387337 3379 tablet_replica.cc:330] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c18d9365982d49178ddd5361ac461f0a: stopping tablet replica
I20260925 00:48:29.388707 3380 tablet_replica.cc:330] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: stopping tablet replica
I20260925 00:48:29.388855 3380 raft_consensus.cc:2229] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:29.389041 3380 raft_consensus.cc:2258] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:29.389669 3380 ts_tablet_manager.cc:1905] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:29.389883 3381 tablet_replica.cc:330] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: stopping tablet replica
I20260925 00:48:29.390027 3381 raft_consensus.cc:2229] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:29.390146 3381 raft_consensus.cc:2258] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:29.390704 3381 ts_tablet_manager.cc:1905] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:29.392515 3380 ts_tablet_manager.cc:1918] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:48:29.392594 3380 log.cc:1196] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/61ffe9c9ddfd4a2c82e977e60d81d010
I20260925 00:48:29.392858 3380 ts_tablet_manager.cc:1939] T 61ffe9c9ddfd4a2c82e977e60d81d010 P c3e8e163f7094e1dbcf37c6bce262e76: Deleting consensus metadata
I20260925 00:48:29.393551 3381 ts_tablet_manager.cc:1918] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 00:48:29.393631 3381 log.cc:1196] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/61ffe9c9ddfd4a2c82e977e60d81d010
I20260925 00:48:29.393841 3381 ts_tablet_manager.cc:1939] T 61ffe9c9ddfd4a2c82e977e60d81d010 P 17c89777e0f04eb2835189502d22523d: Deleting consensus metadata
I20260925 00:48:29.393790 2720 catalog_manager.cc:4841] TS c3e8e163f7094e1dbcf37c6bce262e76 (127.0.249.67:37115): tablet 61ffe9c9ddfd4a2c82e977e60d81d010 (table test-table [id=63e7b89c39374f94acaf165865b91063]) successfully deleted
I20260925 00:48:29.394446 2719 catalog_manager.cc:4841] TS 17c89777e0f04eb2835189502d22523d (127.0.249.66:39843): tablet 61ffe9c9ddfd4a2c82e977e60d81d010 (table test-table [id=63e7b89c39374f94acaf165865b91063]) successfully deleted
I20260925 00:48:29.395565 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 2945
W20260925 00:48:29.395670 2720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:45507 (error 108)
I20260925 00:48:29.402364 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:29.397096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45507 (local address 127.0.0.1:46542)
0925 00:48:29.402132 (+ 5036us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:29.402224 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4935,"thread_start_us":81,"threads_started":1}
I20260925 00:48:29.402456 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 3085
W20260925 00:48:29.402527 2720 catalog_manager.cc:4584] TS c18d9365982d49178ddd5361ac461f0a (127.0.249.65:45507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61ffe9c9ddfd4a2c82e977e60d81d010: Network error: Client connection negotiation failed: client connection to 127.0.249.65:45507: connect: Connection refused (error 111)
I20260925 00:48:29.409044 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 2701
2026-09-25T00:48:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6180 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:29 dist-test-slave-05m0 krb5kdc[3386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:29 dist-test-slave-05m0 krb5kdc[3386](info): set up 2 sockets
Sep 25 00:48:29 dist-test-slave-05m0 krb5kdc[3386](info): commencing operation
krb5kdc: starting...
W20260925 00:48:30.449364 997 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:30 dist-test-slave-05m0 krb5kdc[3386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:30.568638 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:45121
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:45121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:30.678555 3399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:30.678848 3399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:30.678925 3399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:30.682933 3399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:30.683015 3399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:30.683054 3399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:30.683120 3399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:30.683158 3399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:30.688457 3399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:45121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:45121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:30.689528 3399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:30.690546 3399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:30.698779 3407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:30.698779 3405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:30.698792 3404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:30.698884 3399 server_base.cc:1034] running on GCE node
I20260925 00:48:30.699540 3399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:30.700572 3399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:30.701725 3399 hybrid_clock.cc:648] HybridClock initialized: now 1790297310701691 us; error 51 us; skew 500 ppm
Sep 25 00:48:30 dist-test-slave-05m0 krb5kdc[3386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:30.704420 3399 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:30.705271 3399 webserver.cc:460] Webserver started at http://127.0.249.126:33107/ using document root <none> and password file <none>
I20260925 00:48:30.705885 3399 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:30.705987 3399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:30.706213 3399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:30.708280 3399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "ec71524a5e6b4a5c8bf19fda8b53977e"
format_stamp: "Formatted at 2026-09-25 00:48:30 on dist-test-slave-05m0"
server_key: "3a69ca8723ea453f3bce9c0ae1e2aca9"
server_key_iv: "07dac815109b35012901eb9daf03fffa"
server_key_version: "encryptionkey@0"
I20260925 00:48:30.708858 3399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "ec71524a5e6b4a5c8bf19fda8b53977e"
format_stamp: "Formatted at 2026-09-25 00:48:30 on dist-test-slave-05m0"
server_key: "3a69ca8723ea453f3bce9c0ae1e2aca9"
server_key_iv: "07dac815109b35012901eb9daf03fffa"
server_key_version: "encryptionkey@0"
I20260925 00:48:30.712399 3399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:48:30.714797 3414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:30.715999 3399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:30.716163 3399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "ec71524a5e6b4a5c8bf19fda8b53977e"
format_stamp: "Formatted at 2026-09-25 00:48:30 on dist-test-slave-05m0"
server_key: "3a69ca8723ea453f3bce9c0ae1e2aca9"
server_key_iv: "07dac815109b35012901eb9daf03fffa"
server_key_version: "encryptionkey@0"
I20260925 00:48:30.716272 3399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:30.731274 3399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:30.734467 3399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:30.734688 3399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:30.742388 3399 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:45121
I20260925 00:48:30.742416 3466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:45121 every 8 connection(s)
I20260925 00:48:30.743382 3399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:30.745065 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 3399
I20260925 00:48:30.745184 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:30.745507 997 external_mini_cluster.cc:1428] Setting key 1043e0ad09c06f1511e4b620cbc88683
I20260925 00:48:30.746652 3467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:30.752279 3467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e: Bootstrap starting.
Sep 25 00:48:30 dist-test-slave-05m0 krb5kdc[3386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:30.754724 3467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:30.755568 3467 log.cc:824] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:30.757710 3467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e: No bootstrap required, opened a new log
I20260925 00:48:30.759747 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:30.747009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.0.249.126:45121)
0925 00:48:30.747457 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:30.747473 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:30.747508 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:48:30.748148 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:30.748164 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:30.748487 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:30.748833 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:30.749704 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:30.750709 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:30.751618 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:30.752119 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:30.752296 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:30.754849 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:30.754856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:30.754876 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:30.754912 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:30.756802 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:30.757266 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:30.757271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:30.757276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:30.757383 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:30.757734 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:30.757738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:30.757740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:30.758190 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:30.758415 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:30.758748 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 00:48:30.759037 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":150,"threads_started":1}
I20260925 00:48:30.761669 3467 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec71524a5e6b4a5c8bf19fda8b53977e" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45121 } }
I20260925 00:48:30.761979 3467 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:30.762092 3467 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec71524a5e6b4a5c8bf19fda8b53977e, State: Initialized, Role: FOLLOWER
I20260925 00:48:30.762545 3467 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec71524a5e6b4a5c8bf19fda8b53977e" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45121 } }
I20260925 00:48:30.762673 3467 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:30.762719 3467 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:30.762790 3467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:30.763705 3467 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec71524a5e6b4a5c8bf19fda8b53977e" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45121 } }
I20260925 00:48:30.764009 3467 leader_election.cc:302] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec71524a5e6b4a5c8bf19fda8b53977e; no voters:
I20260925 00:48:30.764258 3467 leader_election.cc:288] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:30.764401 3472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:30.764688 3472 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [term 1 LEADER]: Becoming Leader. State: Replica: ec71524a5e6b4a5c8bf19fda8b53977e, State: Running, Role: LEADER
I20260925 00:48:30.765074 3472 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec71524a5e6b4a5c8bf19fda8b53977e" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45121 } }
I20260925 00:48:30.765647 3467 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:30.766938 3473 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec71524a5e6b4a5c8bf19fda8b53977e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec71524a5e6b4a5c8bf19fda8b53977e" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45121 } } }
I20260925 00:48:30.766973 3474 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec71524a5e6b4a5c8bf19fda8b53977e. Latest consensus state: current_term: 1 leader_uuid: "ec71524a5e6b4a5c8bf19fda8b53977e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec71524a5e6b4a5c8bf19fda8b53977e" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45121 } } }
I20260925 00:48:30.767091 3473 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:30.767094 3474 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:30.767448 3481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:30.770548 3481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:30.775398 3481 catalog_manager.cc:1347] Generated new cluster ID: 82818d39da824c95b4e822c1d66da2d4
I20260925 00:48:30.775470 3481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:30.791033 3481 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:30.791978 3481 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:30.796166 3481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec71524a5e6b4a5c8bf19fda8b53977e: Generated new TSK 0
I20260925 00:48:30.796824 3481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:48:30.832363 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45121
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:30.942615 3493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:30.942870 3493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:30.942914 3493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:30.947203 3493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:30.947263 3493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:30.947322 3493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:30.952646 3493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:30.953692 3493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:30.954638 3493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:30.963959 3498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:30.963967 3501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:30.963959 3499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:30.964206 3493 server_base.cc:1034] running on GCE node
I20260925 00:48:30.964599 3493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:30.965256 3493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:30.966403 3493 hybrid_clock.cc:648] HybridClock initialized: now 1790297310966367 us; error 55 us; skew 500 ppm
Sep 25 00:48:30 dist-test-slave-05m0 krb5kdc[3386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:30.969182 3493 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:30.970055 3493 webserver.cc:460] Webserver started at http://127.0.249.65:39447/ using document root <none> and password file <none>
I20260925 00:48:30.970608 3493 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:30.970713 3493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:30.970988 3493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:30.972975 3493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "38a60e81b93b42e7a502c9715a7d8809"
format_stamp: "Formatted at 2026-09-25 00:48:30 on dist-test-slave-05m0"
server_key: "2fa8402910e0a6dee8896a3eec824612"
server_key_iv: "69f20887abd2e4397ec77240d798804d"
server_key_version: "encryptionkey@0"
I20260925 00:48:30.973611 3493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "38a60e81b93b42e7a502c9715a7d8809"
format_stamp: "Formatted at 2026-09-25 00:48:30 on dist-test-slave-05m0"
server_key: "2fa8402910e0a6dee8896a3eec824612"
server_key_iv: "69f20887abd2e4397ec77240d798804d"
server_key_version: "encryptionkey@0"
I20260925 00:48:30.977113 3493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 00:48:30.979540 3508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:30.980727 3493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:30.980861 3493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "38a60e81b93b42e7a502c9715a7d8809"
format_stamp: "Formatted at 2026-09-25 00:48:30 on dist-test-slave-05m0"
server_key: "2fa8402910e0a6dee8896a3eec824612"
server_key_iv: "69f20887abd2e4397ec77240d798804d"
server_key_version: "encryptionkey@0"
I20260925 00:48:30.980967 3493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:30.999361 3493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:31.002461 3493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:31.002676 3493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:31.003262 3493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:31.004078 3493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:31.004146 3493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:31.004215 3493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:31.004264 3493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:31.013092 3493 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:45071
I20260925 00:48:31.013120 3621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:45071 every 8 connection(s)
I20260925 00:48:31.014017 3493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:31.018239 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 3493
I20260925 00:48:31.018406 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:31.018738 997 external_mini_cluster.cc:1428] Setting key 05826a033aca8cf4c2a34014c6a86c38
Sep 25 00:48:31 dist-test-slave-05m0 krb5kdc[3386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:48:31.027959 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:31.015762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:41851 (local address 127.0.249.126:45121)
0925 00:48:31.015916 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:31.015921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:31.016808 (+ 887us) server_negotiation.cc:407] Connection header received
0925 00:48:31.018359 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:31.018364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:31.018454 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:31.018570 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:31.019839 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.020479 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.021681 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.021919 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.022031 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:31.024423 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:31.024426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:31.024428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:31.024450 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:31.025965 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:31.026454 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:31.026458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:31.026460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:31.026544 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:31.026867 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:31.026871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:31.026873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:31.027049 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:31.027152 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:31.027654 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 00:48:31.027808 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 00:48:31.028673 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:31.016228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:45121 (local address 127.0.249.65:41851)
0925 00:48:31.016653 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:31.016695 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:31.018124 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:31.018721 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:31.018728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:31.019135 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:31.019696 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:31.019710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.020617 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.020620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:31.021561 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:31.021571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.022071 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.022075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:31.022255 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:31.022874 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:31.022889 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:31.024259 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:31.026085 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:31.026091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:31.026105 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:31.026349 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:31.026634 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:31.026639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:31.026640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:31.026771 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:31.027149 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:31.027156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:31.027389 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 00:48:31.027580 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 00:48:31.027834 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
I20260925 00:48:31.029870 3622 heartbeater.cc:347] Connected to a master server at 127.0.249.126:45121
I20260925 00:48:31.030107 3622 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
I20260925 00:48:31.030632 3622 heartbeater.cc:511] Master 127.0.249.126:45121 requested a full tablet report, sending...
I20260925 00:48:31.032261 3431 ts_manager.cc:194] Registered new tserver with Master: 38a60e81b93b42e7a502c9715a7d8809 (127.0.249.65:45071)
I20260925 00:48:31.033375 3431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:41851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:31.046941 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45121
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:31.156740 3627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:31.157007 3627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:31.157058 3627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:31.161526 3627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:31.161586 3627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:31.161670 3627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:31.167151 3627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:31.168306 3627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:31.169368 3627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:31.179365 3632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:31.179358 3635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:31.179363 3633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:31.179737 3627 server_base.cc:1034] running on GCE node
I20260925 00:48:31.180125 3627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:31.180759 3627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:31.181914 3627 hybrid_clock.cc:648] HybridClock initialized: now 1790297311181892 us; error 31 us; skew 500 ppm
Sep 25 00:48:31 dist-test-slave-05m0 krb5kdc[3386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:31.184864 3627 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:31.185796 3627 webserver.cc:460] Webserver started at http://127.0.249.66:35541/ using document root <none> and password file <none>
I20260925 00:48:31.186359 3627 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:31.186450 3627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:31.186661 3627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:31.188628 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "984467e056954a34bc020efe605b6282"
format_stamp: "Formatted at 2026-09-25 00:48:31 on dist-test-slave-05m0"
server_key: "6024f942d5cc55ae15b76a69601b956f"
server_key_iv: "58612a6362857ef550fa1b0cfe7671ac"
server_key_version: "encryptionkey@0"
I20260925 00:48:31.189195 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "984467e056954a34bc020efe605b6282"
format_stamp: "Formatted at 2026-09-25 00:48:31 on dist-test-slave-05m0"
server_key: "6024f942d5cc55ae15b76a69601b956f"
server_key_iv: "58612a6362857ef550fa1b0cfe7671ac"
server_key_version: "encryptionkey@0"
I20260925 00:48:31.192967 3627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:48:31.195462 3642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:31.196635 3627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:31.196763 3627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "984467e056954a34bc020efe605b6282"
format_stamp: "Formatted at 2026-09-25 00:48:31 on dist-test-slave-05m0"
server_key: "6024f942d5cc55ae15b76a69601b956f"
server_key_iv: "58612a6362857ef550fa1b0cfe7671ac"
server_key_version: "encryptionkey@0"
I20260925 00:48:31.196873 3627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:31.207732 3627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:31.210930 3627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:31.211145 3627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:31.211723 3627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:31.212525 3627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:31.212594 3627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:31.212668 3627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:31.212704 3627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:31.222222 3627 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:39795
I20260925 00:48:31.222239 3755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:39795 every 8 connection(s)
I20260925 00:48:31.223150 3627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
Sep 25 00:48:31 dist-test-slave-05m0 krb5kdc[3386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:31.233129 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 3627
I20260925 00:48:31.233265 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:31.233606 997 external_mini_cluster.cc:1428] Setting key 4a0ed368ffe67f843f9d40434a31bf45
I20260925 00:48:31.237998 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:31.225068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:51695 (local address 127.0.249.126:45121)
0925 00:48:31.225227 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:31.225233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:31.226299 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 00:48:31.227641 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:31.227644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:31.227697 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:31.227785 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:31.228924 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.229585 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.230686 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.230915 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.231007 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:31.233673 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:31.233696 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:31.233701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:31.233735 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:31.235530 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:31.236126 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:31.236131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:31.236133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:31.236229 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:31.236625 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:31.236631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:31.236633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:31.236923 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:31.237038 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:31.237658 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 00:48:31.237802 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:48:31.238912 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:31.225676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:45121 (local address 127.0.249.66:51695)
0925 00:48:31.226144 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:31.226187 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:31.227420 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:31.227940 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:31.227947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:31.228301 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:31.228769 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:31.228783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.229699 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.229702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:31.230559 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:31.230567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.231060 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.231063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:31.231201 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:31.231773 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:31.231788 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:31.233482 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:31.235689 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:31.235695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:31.235712 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:31.236009 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:31.236341 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:31.236344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:31.236347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:31.236516 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:31.237038 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:31.237046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:31.237330 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 00:48:31.237613 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 00:48:31.237984 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
I20260925 00:48:31.240192 3756 heartbeater.cc:347] Connected to a master server at 127.0.249.126:45121
I20260925 00:48:31.240479 3756 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:31.241115 3756 heartbeater.cc:511] Master 127.0.249.126:45121 requested a full tablet report, sending...
I20260925 00:48:31.242324 3431 ts_manager.cc:194] Registered new tserver with Master: 984467e056954a34bc020efe605b6282 (127.0.249.66:39795)
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:48:31.243170 3431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:51695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:31.260366 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45121
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:31.373483 3761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:31.373760 3761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:31.373804 3761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:31.378278 3761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:31.378346 3761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:31.378458 3761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:31.383783 3761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:31.384837 3761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:31.385944 3761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:31.395174 3766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:31.395211 3769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:31.395197 3767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:31.395654 3761 server_base.cc:1034] running on GCE node
I20260925 00:48:31.396083 3761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:31.396689 3761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:31.397867 3761 hybrid_clock.cc:648] HybridClock initialized: now 1790297311397858 us; error 30 us; skew 500 ppm
Sep 25 00:48:31 dist-test-slave-05m0 krb5kdc[3386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:31.400666 3761 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:31.401558 3761 webserver.cc:460] Webserver started at http://127.0.249.67:39149/ using document root <none> and password file <none>
I20260925 00:48:31.402127 3761 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:31.402213 3761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:31.402438 3761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:31.404449 3761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "ad9215fc26844fd0bb9bf2115e2efd6e"
format_stamp: "Formatted at 2026-09-25 00:48:31 on dist-test-slave-05m0"
server_key: "114e06ef41342b87b70b454abd160afb"
server_key_iv: "81dd9f3e26f604eb288d92939911cbcd"
server_key_version: "encryptionkey@0"
I20260925 00:48:31.405017 3761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "ad9215fc26844fd0bb9bf2115e2efd6e"
format_stamp: "Formatted at 2026-09-25 00:48:31 on dist-test-slave-05m0"
server_key: "114e06ef41342b87b70b454abd160afb"
server_key_iv: "81dd9f3e26f604eb288d92939911cbcd"
server_key_version: "encryptionkey@0"
I20260925 00:48:31.408802 3761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:48:31.411424 3776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:31.412618 3761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:31.412758 3761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "ad9215fc26844fd0bb9bf2115e2efd6e"
format_stamp: "Formatted at 2026-09-25 00:48:31 on dist-test-slave-05m0"
server_key: "114e06ef41342b87b70b454abd160afb"
server_key_iv: "81dd9f3e26f604eb288d92939911cbcd"
server_key_version: "encryptionkey@0"
I20260925 00:48:31.412871 3761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:31.437867 3761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:31.441926 3761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:31.442178 3761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:31.442878 3761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:31.443854 3761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:31.443940 3761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:31.444024 3761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:31.444087 3761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:31.454779 3761 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:35443
I20260925 00:48:31.454792 3889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:35443 every 8 connection(s)
I20260925 00:48:31.455852 3761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:31.456588 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 3761
I20260925 00:48:31.456697 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:31.457007 997 external_mini_cluster.cc:1428] Setting key 3b642cc56b1e01ad9d216f60973c20d1
Sep 25 00:48:31 dist-test-slave-05m0 krb5kdc[3386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:31.470492 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:31.458076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:33351 (local address 127.0.249.126:45121)
0925 00:48:31.458203 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:31.458207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:31.459462 (+ 1255us) server_negotiation.cc:407] Connection header received
0925 00:48:31.460826 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:31.460830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:31.460897 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:31.460973 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:31.462189 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.462721 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.463904 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.464148 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.464260 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:31.466827 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:31.466831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:31.466834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:31.466875 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:31.468397 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:31.468895 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:31.468900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:31.468901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:31.468980 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:31.469309 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:31.469313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:31.469315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:31.469547 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:31.469646 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:31.470203 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 00:48:31.470332 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:31.471374 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:31.458709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:45121 (local address 127.0.249.67:33351)
0925 00:48:31.459294 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:31.459354 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:31.460625 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:31.461146 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:31.461153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:31.461544 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:31.462026 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:31.462040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.462845 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.462848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:31.463773 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:31.463782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:31.464299 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:31.464303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:31.464479 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:31.465044 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:31.465059 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:31.466638 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:31.468513 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:31.468518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:31.468534 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:31.468788 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:31.469069 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:31.469073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:31.469075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:31.469212 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:31.469656 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:31.469664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:31.469958 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 00:48:31.470165 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 00:48:31.470506 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":165,"threads_started":1}
I20260925 00:48:31.472532 3890 heartbeater.cc:347] Connected to a master server at 127.0.249.126:45121
I20260925 00:48:31.472744 3890 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:31.473191 3890 heartbeater.cc:511] Master 127.0.249.126:45121 requested a full tablet report, sending...
I20260925 00:48:31.474231 3431 ts_manager.cc:194] Registered new tserver with Master: ad9215fc26844fd0bb9bf2115e2efd6e (127.0.249.67:35443)
I20260925 00:48:31.474804 3431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:33351
I20260925 00:48:31.483213 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 00:48:31.496397 3470 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:31.494928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.249.126:45121)
0925 00:48:31.495110 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:31.495117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:31.495233 (+ 116us) server_negotiation.cc:407] Connection header received
0925 00:48:31.495351 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:31.495354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:31.495398 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:31.495484 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:31.496252 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38312: BlockingRecv error: recv got EOF from 127.0.0.1:38312 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 00:48:31.497491 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 3493
I20260925 00:48:31.502986 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 3627
I20260925 00:48:31.508000 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 3761
I20260925 00:48:31.513298 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 3399
2026-09-25T00:48:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2104 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:31 dist-test-slave-05m0 krb5kdc[3902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:31 dist-test-slave-05m0 krb5kdc[3902](info): set up 2 sockets
Sep 25 00:48:31 dist-test-slave-05m0 krb5kdc[3902](info): commencing operation
krb5kdc: starting...
W20260925 00:48:32.585543 997 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:32 dist-test-slave-05m0 krb5kdc[3902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:32.702397 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:33599
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:33599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:32.809309 3915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:32.809669 3915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:32.809752 3915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:32.813814 3915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:32.813889 3915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:32.813926 3915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:32.813982 3915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:32.814019 3915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:32.819130 3915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:33599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:33599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:32.820117 3915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:32.821069 3915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:32.829216 3923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:32.829238 3920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:32.829252 3915 server_base.cc:1034] running on GCE node
W20260925 00:48:32.829212 3921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:32.830054 3915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:32.831125 3915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:32.832293 3915 hybrid_clock.cc:648] HybridClock initialized: now 1790297312832276 us; error 29 us; skew 500 ppm
Sep 25 00:48:32 dist-test-slave-05m0 krb5kdc[3902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:32.841529 3915 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:32.842406 3915 webserver.cc:460] Webserver started at http://127.0.249.126:36595/ using document root <none> and password file <none>
I20260925 00:48:32.842998 3915 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:32.843096 3915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:32.843319 3915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:32.845321 3915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "c870a94478754ec7a5ddea75541021f3"
format_stamp: "Formatted at 2026-09-25 00:48:32 on dist-test-slave-05m0"
server_key: "f320e37a0bea5ee7d5587e6d75bb5d9c"
server_key_iv: "f2e18b517115f4e0c2d11831d8a2ff50"
server_key_version: "encryptionkey@0"
I20260925 00:48:32.845959 3915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "c870a94478754ec7a5ddea75541021f3"
format_stamp: "Formatted at 2026-09-25 00:48:32 on dist-test-slave-05m0"
server_key: "f320e37a0bea5ee7d5587e6d75bb5d9c"
server_key_iv: "f2e18b517115f4e0c2d11831d8a2ff50"
server_key_version: "encryptionkey@0"
I20260925 00:48:32.849690 3915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:48:32.851964 3930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:32.853104 3915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:32.853246 3915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "c870a94478754ec7a5ddea75541021f3"
format_stamp: "Formatted at 2026-09-25 00:48:32 on dist-test-slave-05m0"
server_key: "f320e37a0bea5ee7d5587e6d75bb5d9c"
server_key_iv: "f2e18b517115f4e0c2d11831d8a2ff50"
server_key_version: "encryptionkey@0"
I20260925 00:48:32.853358 3915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:32.871822 3915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:32.874773 3915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:32.874979 3915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:32.881896 3915 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:33599
I20260925 00:48:32.882174 3982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:33599 every 8 connection(s)
I20260925 00:48:32.882845 3915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:32.885497 3983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:32.888170 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 3915
I20260925 00:48:32.888293 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:32.888589 997 external_mini_cluster.cc:1428] Setting key d90ac95021c074cdff7254475f9177b6
I20260925 00:48:32.891299 3983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3: Bootstrap starting.
I20260925 00:48:32.893839 3983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:32.894730 3983 log.cc:824] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:32.896998 3983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3: No bootstrap required, opened a new log
Sep 25 00:48:32 dist-test-slave-05m0 krb5kdc[3902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:32.899925 3983 raft_consensus.cc:348] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c870a94478754ec7a5ddea75541021f3" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33599 } }
I20260925 00:48:32.900135 3983 raft_consensus.cc:374] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:32.900205 3983 raft_consensus.cc:729] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c870a94478754ec7a5ddea75541021f3, State: Initialized, Role: FOLLOWER
I20260925 00:48:32.900625 3983 consensus_queue.cc:260] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c870a94478754ec7a5ddea75541021f3" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33599 } }
I20260925 00:48:32.900771 3983 raft_consensus.cc:388] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:32.900854 3983 raft_consensus.cc:482] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:32.900974 3983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:32.901862 3983 raft_consensus.cc:504] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c870a94478754ec7a5ddea75541021f3" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33599 } }
I20260925 00:48:32.902235 3983 leader_election.cc:302] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c870a94478754ec7a5ddea75541021f3; no voters:
I20260925 00:48:32.902578 3983 leader_election.cc:288] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:32.902671 3988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:32.902892 3988 raft_consensus.cc:686] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [term 1 LEADER]: Becoming Leader. State: Replica: c870a94478754ec7a5ddea75541021f3, State: Running, Role: LEADER
I20260925 00:48:32.903252 3988 consensus_queue.cc:237] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c870a94478754ec7a5ddea75541021f3" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33599 } }
I20260925 00:48:32.903790 3983 sys_catalog.cc:564] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:32.903954 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:32.889887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.0.249.126:33599)
0925 00:48:32.890299 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:32.890314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:32.890350 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:48:32.890957 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:32.890973 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:32.891282 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:32.891608 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:32.892485 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:32.893531 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:32.894704 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:32.895309 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:32.895489 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:32.898184 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:32.898189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:32.898201 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:32.898226 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:32.900615 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:32.901094 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:32.901100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:32.901106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:32.901222 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:32.901563 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:32.901567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:32.901569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:32.901947 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:32.902151 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:32.902445 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 00:48:32.902933 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260925 00:48:32.905642 3990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c870a94478754ec7a5ddea75541021f3. Latest consensus state: current_term: 1 leader_uuid: "c870a94478754ec7a5ddea75541021f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c870a94478754ec7a5ddea75541021f3" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33599 } } }
I20260925 00:48:32.906466 3989 sys_catalog.cc:455] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c870a94478754ec7a5ddea75541021f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c870a94478754ec7a5ddea75541021f3" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33599 } } }
I20260925 00:48:32.906564 3989 sys_catalog.cc:458] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:32.905843 3990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:32.907042 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 00:48:32.908870 4002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:48:32.908942 4002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:48:32.909018 4003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:32.909909 4003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:32.915241 4003 catalog_manager.cc:1347] Generated new cluster ID: 44861246bd924c849c83d72657bd33f3
I20260925 00:48:32.915299 4003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:32.932323 4003 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:32.933329 4003 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:32.937769 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:32.928962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.0.249.126:33599)
0925 00:48:32.929101 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:32.929110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:32.929368 (+ 258us) server_negotiation.cc:407] Connection header received
0925 00:48:32.929575 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:32.929578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:32.929622 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:32.929710 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:32.930227 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:32.930748 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:32.931755 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:32.932072 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:32.932184 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:32.936642 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:32.936646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:32.936648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:32.937230 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:32.937276 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:32.937297 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:32.937414 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:32.937545 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 00:48:32.937627 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:48:32.943459 4003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c870a94478754ec7a5ddea75541021f3: Generated new TSK 0
I20260925 00:48:32.944243 4003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:48:32.953090 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 3915
2026-09-25T00:48:32Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1436 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:33 dist-test-slave-05m0 krb5kdc[4016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:33 dist-test-slave-05m0 krb5kdc[4016](info): set up 2 sockets
Sep 25 00:48:33 dist-test-slave-05m0 krb5kdc[4016](info): commencing operation
krb5kdc: starting...
W20260925 00:48:34.017634 997 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:34 dist-test-slave-05m0 krb5kdc[4016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:34.136477 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:43319
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:43319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:34.248344 4029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:34.248617 4029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:34.248661 4029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:34.253084 4029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:34.253152 4029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:34.253172 4029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:34.253188 4029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:34.253204 4029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:34.258628 4029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:43319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:43319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:34.259615 4029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:34.260597 4029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:34.268978 4035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:34.268978 4037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:34.269008 4034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:34.269613 4029 server_base.cc:1034] running on GCE node
I20260925 00:48:34.270192 4029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:34.271318 4029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:34.272529 4029 hybrid_clock.cc:648] HybridClock initialized: now 1790297314272497 us; error 51 us; skew 500 ppm
Sep 25 00:48:34 dist-test-slave-05m0 krb5kdc[4016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:34.275300 4029 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:34.276260 4029 webserver.cc:460] Webserver started at http://127.0.249.126:45175/ using document root <none> and password file <none>
I20260925 00:48:34.276868 4029 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:34.276971 4029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:34.277215 4029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:34.279342 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "588d732cd8c345a5ab3669b45181ca3d"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "65ea78a5ae3817f38288debec44da673"
server_key_iv: "9cefb299f8e0bd14f6f3da12eedef200"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.279951 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "588d732cd8c345a5ab3669b45181ca3d"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "65ea78a5ae3817f38288debec44da673"
server_key_iv: "9cefb299f8e0bd14f6f3da12eedef200"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.283946 4029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 00:48:34.286527 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:34.287825 4029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:34.287990 4029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "588d732cd8c345a5ab3669b45181ca3d"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "65ea78a5ae3817f38288debec44da673"
server_key_iv: "9cefb299f8e0bd14f6f3da12eedef200"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.288110 4029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:34.299077 4029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:34.302192 4029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:34.302412 4029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:34.309659 4029 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:43319
I20260925 00:48:34.309684 4096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:43319 every 8 connection(s)
I20260925 00:48:34.310672 4029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:34.312503 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 4029
I20260925 00:48:34.312599 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:34.312876 997 external_mini_cluster.cc:1428] Setting key 4fc0528f84123dd9a8a2f494ee678c59
I20260925 00:48:34.313735 4097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:34.320518 4097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d: Bootstrap starting.
Sep 25 00:48:34 dist-test-slave-05m0 krb5kdc[4016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:34.323221 4097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:34.324191 4097 log.cc:824] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:34.326511 4097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d: No bootstrap required, opened a new log
I20260925 00:48:34.328274 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:34.314330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.0.249.126:43319)
0925 00:48:34.314764 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:34.314783 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:34.314826 (+ 43us) server_negotiation.cc:407] Connection header received
0925 00:48:34.315710 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:34.315734 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:34.316161 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:34.316527 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:34.317214 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.318722 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.319563 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.320221 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.320379 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:34.323032 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:34.323042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:34.323060 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:34.323099 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:34.324999 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:34.325433 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:34.325438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:34.325444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:34.325587 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:34.325834 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:34.325837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:34.325839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:34.326331 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:34.326525 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:34.326760 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 00:48:34.327081 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":122,"threads_started":1}
I20260925 00:48:34.330303 4097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588d732cd8c345a5ab3669b45181ca3d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43319 } }
I20260925 00:48:34.330611 4097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:34.330719 4097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588d732cd8c345a5ab3669b45181ca3d, State: Initialized, Role: FOLLOWER
I20260925 00:48:34.331110 4097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588d732cd8c345a5ab3669b45181ca3d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43319 } }
I20260925 00:48:34.331228 4097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:34.331270 4097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:34.331342 4097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:34.332209 4097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588d732cd8c345a5ab3669b45181ca3d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43319 } }
I20260925 00:48:34.332497 4097 leader_election.cc:302] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 588d732cd8c345a5ab3669b45181ca3d; no voters:
I20260925 00:48:34.332741 4097 leader_election.cc:288] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:34.332893 4102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:34.333134 4102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [term 1 LEADER]: Becoming Leader. State: Replica: 588d732cd8c345a5ab3669b45181ca3d, State: Running, Role: LEADER
I20260925 00:48:34.333508 4102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588d732cd8c345a5ab3669b45181ca3d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43319 } }
I20260925 00:48:34.334038 4097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:34.335191 4104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588d732cd8c345a5ab3669b45181ca3d. Latest consensus state: current_term: 1 leader_uuid: "588d732cd8c345a5ab3669b45181ca3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588d732cd8c345a5ab3669b45181ca3d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43319 } } }
I20260925 00:48:34.335191 4103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "588d732cd8c345a5ab3669b45181ca3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588d732cd8c345a5ab3669b45181ca3d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43319 } } }
I20260925 00:48:34.335346 4104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:34.335399 4103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:34.335780 4111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:34.338846 4111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:34.344501 4111 catalog_manager.cc:1347] Generated new cluster ID: fa14d4c4614c4fb29f4663b146d1be64
I20260925 00:48:34.344583 4111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:34.359277 4111 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:34.360263 4111 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:34.368086 4111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 588d732cd8c345a5ab3669b45181ca3d: Generated new TSK 0
I20260925 00:48:34.368908 4111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:48:34.402803 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43319
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:34.513288 4123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:34.513581 4123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:34.513624 4123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:34.517597 4123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:34.517654 4123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:34.517711 4123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:34.522941 4123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:34.523883 4123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:34.524837 4123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:34.534085 4128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:34.534164 4123 server_base.cc:1034] running on GCE node
W20260925 00:48:34.534126 4131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:34.534080 4129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:34.534795 4123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:34.535480 4123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:34.536656 4123 hybrid_clock.cc:648] HybridClock initialized: now 1790297314536642 us; error 24 us; skew 500 ppm
Sep 25 00:48:34 dist-test-slave-05m0 krb5kdc[4016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:34.539300 4123 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:34.540061 4123 webserver.cc:460] Webserver started at http://127.0.249.65:38973/ using document root <none> and password file <none>
I20260925 00:48:34.540555 4123 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:34.540617 4123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:34.540772 4123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:34.542709 4123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "7ab8464b376f4d9993016e0c17d877b2"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "b943627f71ceeeceebcb423e2c360882"
server_key_iv: "7c4bfb11fcc7895d8578aa698f374e91"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.543216 4123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "7ab8464b376f4d9993016e0c17d877b2"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "b943627f71ceeeceebcb423e2c360882"
server_key_iv: "7c4bfb11fcc7895d8578aa698f374e91"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.546696 4123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:48:34.548993 4138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:34.550209 4123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 00:48:34.550326 4123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "7ab8464b376f4d9993016e0c17d877b2"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "b943627f71ceeeceebcb423e2c360882"
server_key_iv: "7c4bfb11fcc7895d8578aa698f374e91"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.550428 4123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:34.566740 4123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:34.570154 4123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:34.570302 4123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:34.570880 4123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:34.571691 4123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:34.571736 4123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:34.571775 4123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:34.571790 4123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:34.580961 4123 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:34291
I20260925 00:48:34.581024 4251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:34291 every 8 connection(s)
I20260925 00:48:34.581954 4123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:34.589042 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 4123
I20260925 00:48:34.589200 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:34.589545 997 external_mini_cluster.cc:1428] Setting key 936948555be4c4e4c1e16814061c22a8
Sep 25 00:48:34 dist-test-slave-05m0 krb5kdc[4016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:34.596551 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:34.583706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:40621 (local address 127.0.249.126:43319)
0925 00:48:34.583840 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:34.583844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:34.584676 (+ 832us) server_negotiation.cc:407] Connection header received
0925 00:48:34.586064 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:34.586067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:34.586113 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:34.586198 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:34.587297 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.587844 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.589003 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.589312 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.589455 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:34.592550 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:34.592555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:34.592558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:34.592589 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:34.594348 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:34.594838 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:34.594844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:34.594849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:34.594957 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:34.595271 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:34.595278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:34.595282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:34.595552 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:34.595688 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:34.596147 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 00:48:34.596296 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:48:34.597193 4254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:34.584085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43319 (local address 127.0.249.65:40621)
0925 00:48:34.584511 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:34.584554 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:34.585862 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:34.586351 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:34.586358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:34.586704 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:34.587171 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:34.587184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.587989 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.587992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:34.588867 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:34.588875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.589484 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.589488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:34.589716 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:34.590349 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:34.590369 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:34.592339 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:34.594466 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:34.594471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:34.594483 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:34.594737 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:34.595049 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:34.595051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:34.595052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:34.595177 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:34.595665 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:34.595670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:34.595895 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 00:48:34.596065 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 00:48:34.596313 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
I20260925 00:48:34.598459 4252 heartbeater.cc:347] Connected to a master server at 127.0.249.126:43319
I20260925 00:48:34.598721 4252 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:34.599313 4252 heartbeater.cc:511] Master 127.0.249.126:43319 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:48:34.600924 4061 ts_manager.cc:194] Registered new tserver with Master: 7ab8464b376f4d9993016e0c17d877b2 (127.0.249.65:34291)
I20260925 00:48:34.602214 4061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:40621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:34.620077 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43319
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:34.728076 4257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:34.728343 4257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:34.728389 4257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:34.732533 4257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:34.732604 4257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:34.732663 4257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:34.738247 4257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:34.739315 4257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:34.740379 4257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:34.749744 4262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:34.749792 4263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:34.749820 4265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:34.749918 4257 server_base.cc:1034] running on GCE node
I20260925 00:48:34.750355 4257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:34.750990 4257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:34.752161 4257 hybrid_clock.cc:648] HybridClock initialized: now 1790297314752145 us; error 25 us; skew 500 ppm
Sep 25 00:48:34 dist-test-slave-05m0 krb5kdc[4016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:34.755014 4257 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:34.755864 4257 webserver.cc:460] Webserver started at http://127.0.249.66:44817/ using document root <none> and password file <none>
I20260925 00:48:34.756424 4257 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:34.756513 4257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:34.756735 4257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:34.758790 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "467625754f3e4d09b1afdf8a293b43f0"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "55e4065838643088e8fe747d99a30498"
server_key_iv: "8763d743017972b181daaf30bd5bb994"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.759343 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "467625754f3e4d09b1afdf8a293b43f0"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "55e4065838643088e8fe747d99a30498"
server_key_iv: "8763d743017972b181daaf30bd5bb994"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.762888 4257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:48:34.765188 4272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:34.766384 4257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:34.766536 4257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "467625754f3e4d09b1afdf8a293b43f0"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "55e4065838643088e8fe747d99a30498"
server_key_iv: "8763d743017972b181daaf30bd5bb994"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.766646 4257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:34.789710 4257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:34.793136 4257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:34.793340 4257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:34.793982 4257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:34.794807 4257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:34.794878 4257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:34.794940 4257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:34.794983 4257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:34.803617 4257 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:41853
I20260925 00:48:34.803640 4385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:41853 every 8 connection(s)
I20260925 00:48:34.804476 4257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:34.808866 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 4257
I20260925 00:48:34.808998 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:34.809299 997 external_mini_cluster.cc:1428] Setting key 7fce2c72124e1aa2c2d45e57b3892eb2
Sep 25 00:48:34 dist-test-slave-05m0 krb5kdc[4016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:48:34.818794 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:34.806353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:59497 (local address 127.0.249.126:43319)
0925 00:48:34.806500 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:34.806504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:34.807378 (+ 874us) server_negotiation.cc:407] Connection header received
0925 00:48:34.808739 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:34.808745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:34.808806 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:34.808908 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:34.810277 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.810963 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.812103 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.812390 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.812472 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:34.815236 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:34.815239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:34.815241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:34.815264 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:34.816703 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:34.817207 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:34.817212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:34.817214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:34.817300 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:34.817664 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:34.817669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:34.817670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:34.817846 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:34.817952 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:34.818488 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 00:48:34.818621 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:48:34.819591 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:34.806784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43319 (local address 127.0.249.66:59497)
0925 00:48:34.807229 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:34.807272 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:34.808526 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:34.809140 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:34.809147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:34.809599 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:34.810131 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:34.810149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.811085 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.811088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:34.811946 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:34.811955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:34.812541 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:34.812547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:34.812763 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:34.813382 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:34.813467 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:34.815065 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:34.816820 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:34.816826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:34.816840 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:34.817104 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:34.817416 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:34.817419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:34.817421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:34.817567 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:34.817951 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:34.817958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:34.818204 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 00:48:34.818430 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 00:48:34.818749 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":99,"threads_started":1}
I20260925 00:48:34.820783 4386 heartbeater.cc:347] Connected to a master server at 127.0.249.126:43319
I20260925 00:48:34.821049 4386 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
I20260925 00:48:34.821630 4386 heartbeater.cc:511] Master 127.0.249.126:43319 requested a full tablet report, sending...
I20260925 00:48:34.822769 4061 ts_manager.cc:194] Registered new tserver with Master: 467625754f3e4d09b1afdf8a293b43f0 (127.0.249.66:41853)
I20260925 00:48:34.823596 4061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:59497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:34.837001 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43319
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:34.947970 4391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:34.948254 4391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:34.948347 4391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:34.953270 4391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:34.953338 4391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:34.953470 4391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:34.958925 4391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:34.959964 4391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:34.960955 4391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:34.970155 4399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:34.970227 4396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:34.970204 4397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:34.970362 4391 server_base.cc:1034] running on GCE node
I20260925 00:48:34.970860 4391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:34.971498 4391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:34.972683 4391 hybrid_clock.cc:648] HybridClock initialized: now 1790297314972649 us; error 45 us; skew 500 ppm
Sep 25 00:48:34 dist-test-slave-05m0 krb5kdc[4016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:34.975472 4391 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:34.976315 4391 webserver.cc:460] Webserver started at http://127.0.249.67:44213/ using document root <none> and password file <none>
I20260925 00:48:34.976858 4391 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:34.976943 4391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:34.977161 4391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:34.979200 4391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "ae3a1836c8df450c922806853077f013"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "3bb0d209fa22fb67213a7efc66ba4bea"
server_key_iv: "241206d082021e2bc8b1d9a52e73bf48"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.979779 4391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "ae3a1836c8df450c922806853077f013"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "3bb0d209fa22fb67213a7efc66ba4bea"
server_key_iv: "241206d082021e2bc8b1d9a52e73bf48"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.983511 4391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:48:34.985985 4406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:34.987133 4391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:34.987269 4391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "ae3a1836c8df450c922806853077f013"
format_stamp: "Formatted at 2026-09-25 00:48:34 on dist-test-slave-05m0"
server_key: "3bb0d209fa22fb67213a7efc66ba4bea"
server_key_iv: "241206d082021e2bc8b1d9a52e73bf48"
server_key_version: "encryptionkey@0"
I20260925 00:48:34.987380 4391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:34.997588 4391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:35.000200 4391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:35.000396 4391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:35.000959 4391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:35.001807 4391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:35.001875 4391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:35.001942 4391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:35.001987 4391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:35.011040 4391 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:38567
I20260925 00:48:35.011066 4519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:38567 every 8 connection(s)
I20260925 00:48:35.011900 4391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:35.013025 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 4391
I20260925 00:48:35.013154 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:35.013507 997 external_mini_cluster.cc:1428] Setting key 119af823d008d14d0b1054d64c9061c0
Sep 25 00:48:35 dist-test-slave-05m0 krb5kdc[4016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:35.025565 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:35.013746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:36515 (local address 127.0.249.126:43319)
0925 00:48:35.013875 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:35.013880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:35.014921 (+ 1041us) server_negotiation.cc:407] Connection header received
0925 00:48:35.016420 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:35.016423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:35.016468 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:35.016537 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:35.017710 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:35.018278 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:35.019586 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:35.019800 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:35.019884 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:35.022306 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:35.022308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:35.022311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:35.022333 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:35.023818 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:35.024246 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:35.024248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:35.024249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:35.024319 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:35.024599 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:35.024601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:35.024602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:35.024762 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:35.024837 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:35.025266 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 00:48:35.025362 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:48:35.026479 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:35.014206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43319 (local address 127.0.249.67:36515)
0925 00:48:35.014746 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:35.014798 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:35.016242 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:35.016674 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:35.016680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:35.017018 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:35.017498 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:35.017511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:35.018406 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:35.018409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:35.019472 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:35.019483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:35.019960 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:35.019963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:35.020118 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:35.020674 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:35.020689 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:35.022160 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:35.023909 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:35.023913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:35.023924 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:35.024167 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:35.024393 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:35.024395 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:35.024397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:35.024513 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:35.024846 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:35.024851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:35.025067 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 00:48:35.025217 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 00:48:35.025518 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260925 00:48:35.027639 4520 heartbeater.cc:347] Connected to a master server at 127.0.249.126:43319
I20260925 00:48:35.027824 4520 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:35.028249 4520 heartbeater.cc:511] Master 127.0.249.126:43319 requested a full tablet report, sending...
I20260925 00:48:35.029143 4061 ts_manager.cc:194] Registered new tserver with Master: ae3a1836c8df450c922806853077f013 (127.0.249.67:38567)
I20260925 00:48:35.029764 4061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:36515
I20260925 00:48:35.039273 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 00:48:35 dist-test-slave-05m0 krb5kdc[4016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297315, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 00:48:35 dist-test-slave-05m0 krb5kdc[4016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297315, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:35.064424 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:35.055998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.0.249.126:43319)
0925 00:48:35.056270 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:35.056276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:35.056473 (+ 197us) server_negotiation.cc:407] Connection header received
0925 00:48:35.056592 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:35.056595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:35.056647 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:35.056737 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:35.057490 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:35.058008 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:35.058842 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:35.059071 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:35.059175 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:35.061567 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:35.061570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:35.061572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:35.061595 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:35.062734 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:35.063161 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:35.063166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:35.063167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:35.063248 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:35.063534 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:35.063537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:35.063539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:35.063758 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:35.063903 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:35.064136 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 00:48:35.064257 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 00:48:35.064867 4061 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47800
I20260925 00:48:35.065690 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 4123
I20260925 00:48:35.071385 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 4257
I20260925 00:48:35.076444 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 4391
I20260925 00:48:35.081786 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 4029
2026-09-25T00:48:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2130 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:35 dist-test-slave-05m0 krb5kdc[4532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:35 dist-test-slave-05m0 krb5kdc[4532](info): set up 2 sockets
Sep 25 00:48:35 dist-test-slave-05m0 krb5kdc[4532](info): commencing operation
krb5kdc: starting...
W20260925 00:48:36.113569 997 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:36 dist-test-slave-05m0 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:36.230526 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:38303
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:38303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:36.337536 4545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:36.337829 4545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:36.337913 4545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:36.341998 4545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:36.342088 4545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:36.342126 4545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:36.342185 4545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:36.342223 4545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:36.347455 4545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:38303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:38303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:36.348546 4545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:36.349584 4545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:36.357897 4550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:36.357888 4553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:36.357888 4551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:36.358440 4545 server_base.cc:1034] running on GCE node
I20260925 00:48:36.359028 4545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:36.360195 4545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:36.361438 4545 hybrid_clock.cc:648] HybridClock initialized: now 1790297316361368 us; error 29 us; skew 500 ppm
Sep 25 00:48:36 dist-test-slave-05m0 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:36.364229 4545 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:36.365034 4545 webserver.cc:460] Webserver started at http://127.0.249.126:46633/ using document root <none> and password file <none>
I20260925 00:48:36.365613 4545 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:36.365675 4545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:36.365834 4545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:36.367897 4545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "92cf445f7c94422188c211268172cf4a"
format_stamp: "Formatted at 2026-09-25 00:48:36 on dist-test-slave-05m0"
server_key: "fffd74519ecfa5ca554235e0005a4244"
server_key_iv: "0b721c05b5121f30d4028ded538f76fc"
server_key_version: "encryptionkey@0"
I20260925 00:48:36.368402 4545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "92cf445f7c94422188c211268172cf4a"
format_stamp: "Formatted at 2026-09-25 00:48:36 on dist-test-slave-05m0"
server_key: "fffd74519ecfa5ca554235e0005a4244"
server_key_iv: "0b721c05b5121f30d4028ded538f76fc"
server_key_version: "encryptionkey@0"
I20260925 00:48:36.371945 4545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 00:48:36.374277 4560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:36.375438 4545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 00:48:36.375590 4545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "92cf445f7c94422188c211268172cf4a"
format_stamp: "Formatted at 2026-09-25 00:48:36 on dist-test-slave-05m0"
server_key: "fffd74519ecfa5ca554235e0005a4244"
server_key_iv: "0b721c05b5121f30d4028ded538f76fc"
server_key_version: "encryptionkey@0"
I20260925 00:48:36.375679 4545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:36.391422 4545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:36.394692 4545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:36.394841 4545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:36.401546 4545 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:38303
I20260925 00:48:36.401623 4612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:38303 every 8 connection(s)
I20260925 00:48:36.402526 4545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:36.405241 4613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:36.406213 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 4545
I20260925 00:48:36.406311 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:36.406657 997 external_mini_cluster.cc:1428] Setting key d5d75e7bb4e58fe07f681fca2a70686e
I20260925 00:48:36.411039 4613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a: Bootstrap starting.
I20260925 00:48:36.413156 4613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:36.413887 4613 log.cc:824] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:48:36 dist-test-slave-05m0 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:36.415685 4613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a: No bootstrap required, opened a new log
I20260925 00:48:36.419369 4613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf445f7c94422188c211268172cf4a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38303 } }
I20260925 00:48:36.419612 4613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:36.419689 4613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cf445f7c94422188c211268172cf4a, State: Initialized, Role: FOLLOWER
I20260925 00:48:36.420087 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:36.408043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.0.249.126:38303)
0925 00:48:36.408451 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:36.408465 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:36.408499 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:48:36.409122 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:36.409137 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:36.409519 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:36.409844 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:36.410626 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.411620 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.412459 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.412920 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.413082 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:36.415567 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:36.415572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:36.415584 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:36.415609 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:36.417245 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:36.417723 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:36.417728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:36.417732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:36.417845 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:36.418126 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:36.418130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:36.418132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:36.418494 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:36.418731 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:36.418957 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 00:48:36.419261 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":131,"threads_started":1}
I20260925 00:48:36.420104 4613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf445f7c94422188c211268172cf4a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38303 } }
I20260925 00:48:36.420292 4613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:36.420380 4613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:36.420501 4613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:36.421334 4613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf445f7c94422188c211268172cf4a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38303 } }
I20260925 00:48:36.421685 4613 leader_election.cc:302] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92cf445f7c94422188c211268172cf4a; no voters:
I20260925 00:48:36.422001 4613 leader_election.cc:288] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:36.422070 4618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:36.422297 4618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [term 1 LEADER]: Becoming Leader. State: Replica: 92cf445f7c94422188c211268172cf4a, State: Running, Role: LEADER
I20260925 00:48:36.422626 4618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf445f7c94422188c211268172cf4a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38303 } }
I20260925 00:48:36.423267 4613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:36.424220 4619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92cf445f7c94422188c211268172cf4a. Latest consensus state: current_term: 1 leader_uuid: "92cf445f7c94422188c211268172cf4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf445f7c94422188c211268172cf4a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38303 } } }
I20260925 00:48:36.424358 4619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:36.424602 4619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92cf445f7c94422188c211268172cf4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf445f7c94422188c211268172cf4a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38303 } } }
I20260925 00:48:36.424674 4619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:36.424758 4627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:36.427772 4627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:36.433121 4627 catalog_manager.cc:1347] Generated new cluster ID: 8eff1925d941499c9bfc8adbe9c93d01
I20260925 00:48:36.433178 4627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:36.447487 4627 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:36.448599 4627 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:36.453542 4627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92cf445f7c94422188c211268172cf4a: Generated new TSK 0
I20260925 00:48:36.454191 4627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:48:36.482700 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:38303
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:36.593889 4639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:36.594170 4639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:36.594245 4639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:36.598402 4639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:36.598487 4639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:36.598577 4639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:36.603677 4639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:38303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:36.604708 4639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:36.605773 4639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:36.615192 4647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:36.615190 4644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:36.615216 4645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:36.615485 4639 server_base.cc:1034] running on GCE node
I20260925 00:48:36.615989 4639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:36.616605 4639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:36.617789 4639 hybrid_clock.cc:648] HybridClock initialized: now 1790297316617777 us; error 33 us; skew 500 ppm
Sep 25 00:48:36 dist-test-slave-05m0 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:36.620630 4639 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:36.621532 4639 webserver.cc:460] Webserver started at http://127.0.249.65:44859/ using document root <none> and password file <none>
I20260925 00:48:36.622092 4639 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:36.622181 4639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:36.622416 4639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:36.624621 4639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "9abbebad2a8242dbb714c6ac64bc47eb"
format_stamp: "Formatted at 2026-09-25 00:48:36 on dist-test-slave-05m0"
server_key: "ee00247e970411dd82f6951be2557203"
server_key_iv: "585d41d5ba78f01996ae642cd82a5cab"
server_key_version: "encryptionkey@0"
I20260925 00:48:36.625192 4639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "9abbebad2a8242dbb714c6ac64bc47eb"
format_stamp: "Formatted at 2026-09-25 00:48:36 on dist-test-slave-05m0"
server_key: "ee00247e970411dd82f6951be2557203"
server_key_iv: "585d41d5ba78f01996ae642cd82a5cab"
server_key_version: "encryptionkey@0"
I20260925 00:48:36.629107 4639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 00:48:36.631462 4654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:36.632686 4639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:48:36.632931 4639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "9abbebad2a8242dbb714c6ac64bc47eb"
format_stamp: "Formatted at 2026-09-25 00:48:36 on dist-test-slave-05m0"
server_key: "ee00247e970411dd82f6951be2557203"
server_key_iv: "585d41d5ba78f01996ae642cd82a5cab"
server_key_version: "encryptionkey@0"
I20260925 00:48:36.633042 4639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:36.666019 4639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:36.669443 4639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:36.669649 4639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:36.670361 4639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:36.671289 4639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:36.671334 4639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:36.671375 4639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:36.671391 4639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:36.680585 4639 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:44771
I20260925 00:48:36.680603 4767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:44771 every 8 connection(s)
I20260925 00:48:36.681557 4639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:36.689230 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 4639
I20260925 00:48:36.689414 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:36.689805 997 external_mini_cluster.cc:1428] Setting key c42a0e54bd2e3bf7a8dcbf31c87f5829
Sep 25 00:48:36 dist-test-slave-05m0 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:36.695974 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:36.683294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:40119 (local address 127.0.249.126:38303)
0925 00:48:36.683445 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:36.683450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:36.684319 (+ 869us) server_negotiation.cc:407] Connection header received
0925 00:48:36.685729 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:36.685731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:36.685775 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:36.685847 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:36.687195 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.687711 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.688896 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.689109 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.689179 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:36.692039 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:36.692042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:36.692045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:36.692076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:36.693846 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:36.694320 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:36.694327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:36.694332 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:36.694422 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:36.694735 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:36.694741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:36.694746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:36.694989 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:36.695100 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:36.695631 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 00:48:36.695776 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 00:48:36.696647 4770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:36.683733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:38303 (local address 127.0.249.65:40119)
0925 00:48:36.684159 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:36.684202 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:36.685539 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:36.685978 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:36.685985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:36.686332 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:36.687046 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:36.687065 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.687876 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.687878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:36.688775 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:36.688785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.689267 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.689273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:36.689524 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:36.690189 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:36.690213 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:36.691882 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:36.693965 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:36.693970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:36.693981 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:36.694225 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:36.694516 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:36.694518 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:36.694519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:36.694644 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:36.695099 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:36.695104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:36.695322 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 00:48:36.695558 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 00:48:36.695809 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260925 00:48:36.697929 4768 heartbeater.cc:347] Connected to a master server at 127.0.249.126:38303
I20260925 00:48:36.698184 4768 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:36.698760 4768 heartbeater.cc:511] Master 127.0.249.126:38303 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:48:36.700346 4577 ts_manager.cc:194] Registered new tserver with Master: 9abbebad2a8242dbb714c6ac64bc47eb (127.0.249.65:44771)
I20260925 00:48:36.701601 4577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:40119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:36.716190 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:38303
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:36.823452 4773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:36.823769 4773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:36.823848 4773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:36.827970 4773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:36.828056 4773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:36.828152 4773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:36.833178 4773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:38303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:36.834192 4773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:36.835170 4773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:36.844461 4778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:36.844441 4781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:36.844533 4773 server_base.cc:1034] running on GCE node
W20260925 00:48:36.844441 4779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:36.845052 4773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:36.845670 4773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:36.846827 4773 hybrid_clock.cc:648] HybridClock initialized: now 1790297316846800 us; error 38 us; skew 500 ppm
Sep 25 00:48:36 dist-test-slave-05m0 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:36.849479 4773 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:36.850245 4773 webserver.cc:460] Webserver started at http://127.0.249.66:45827/ using document root <none> and password file <none>
I20260925 00:48:36.850843 4773 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:36.850934 4773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:36.851161 4773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:36.853080 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "304339d13ec34c80a030e8001ebed587"
format_stamp: "Formatted at 2026-09-25 00:48:36 on dist-test-slave-05m0"
server_key: "a52b33f1f6aecc054282aceb506c2c0d"
server_key_iv: "9adeb39b00d97105736623266e45d500"
server_key_version: "encryptionkey@0"
I20260925 00:48:36.853642 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "304339d13ec34c80a030e8001ebed587"
format_stamp: "Formatted at 2026-09-25 00:48:36 on dist-test-slave-05m0"
server_key: "a52b33f1f6aecc054282aceb506c2c0d"
server_key_iv: "9adeb39b00d97105736623266e45d500"
server_key_version: "encryptionkey@0"
I20260925 00:48:36.857095 4773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:48:36.859467 4788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:36.860615 4773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:48:36.860744 4773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "304339d13ec34c80a030e8001ebed587"
format_stamp: "Formatted at 2026-09-25 00:48:36 on dist-test-slave-05m0"
server_key: "a52b33f1f6aecc054282aceb506c2c0d"
server_key_iv: "9adeb39b00d97105736623266e45d500"
server_key_version: "encryptionkey@0"
I20260925 00:48:36.860849 4773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:36.878396 4773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:36.881603 4773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:36.881814 4773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:36.882368 4773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:36.883145 4773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:36.883221 4773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:36.883312 4773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:36.883363 4773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:36.892092 4773 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:46383
I20260925 00:48:36.892118 4901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:46383 every 8 connection(s)
I20260925 00:48:36.892933 4773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:36.902259 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 4773
I20260925 00:48:36.902395 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:36.902743 997 external_mini_cluster.cc:1428] Setting key 8f0119dbdc84e62f68a886c17a460627
Sep 25 00:48:36 dist-test-slave-05m0 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:36.909695 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:36.894699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:40057 (local address 127.0.249.126:38303)
0925 00:48:36.894832 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:36.894836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:36.895648 (+ 812us) server_negotiation.cc:407] Connection header received
0925 00:48:36.897032 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:36.897036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:36.897088 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:36.897168 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:36.898326 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.899042 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.900796 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.901244 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.901367 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:36.905574 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:36.905578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:36.905581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:36.905606 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:36.907520 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:36.908165 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:36.908168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:36.908170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:36.908258 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:36.908596 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:36.908601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:36.908603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:36.908786 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:36.908890 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:36.909371 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 00:48:36.909530 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:48:36.910463 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:36.895078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:38303 (local address 127.0.249.66:40057)
0925 00:48:36.895501 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:36.895542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:36.896812 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:36.897328 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:36.897335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:36.897739 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:36.898179 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:36.898192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.899192 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.899195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:36.900628 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:36.900642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:36.901444 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:36.901450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:36.901686 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:36.903283 (+ 1597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:36.903304 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:36.905353 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:36.907675 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:36.907681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:36.907696 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:36.908050 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:36.908353 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:36.908357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:36.908359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:36.908500 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:36.908887 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:36.908894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:36.909140 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 00:48:36.909327 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 00:48:36.909656 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260925 00:48:36.911662 4902 heartbeater.cc:347] Connected to a master server at 127.0.249.126:38303
I20260925 00:48:36.911903 4902 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:36.912355 4902 heartbeater.cc:511] Master 127.0.249.126:38303 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:48:36.913347 4577 ts_manager.cc:194] Registered new tserver with Master: 304339d13ec34c80a030e8001ebed587 (127.0.249.66:46383)
I20260925 00:48:36.913949 4577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:40057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:36.930104 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:38303
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:37.038918 4907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:37.039167 4907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:37.039233 4907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:37.043247 4907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:37.043301 4907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:37.043390 4907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:37.048435 4907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:38303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:37.049484 4907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:37.050488 4907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:37.059839 4915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:37.059971 4907 server_base.cc:1034] running on GCE node
W20260925 00:48:37.059826 4913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:37.059867 4912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:37.060431 4907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:37.061043 4907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:37.062197 4907 hybrid_clock.cc:648] HybridClock initialized: now 1790297317062182 us; error 26 us; skew 500 ppm
Sep 25 00:48:37 dist-test-slave-05m0 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:37.065218 4907 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:37.066264 4907 webserver.cc:460] Webserver started at http://127.0.249.67:37875/ using document root <none> and password file <none>
I20260925 00:48:37.066890 4907 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:37.066993 4907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:37.067260 4907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:37.069517 4907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "3f7764946e4a49b8b3f95afaf4742955"
format_stamp: "Formatted at 2026-09-25 00:48:37 on dist-test-slave-05m0"
server_key: "0f4c7ad05bd36cb117a20325e31ccbde"
server_key_iv: "e048cf742d76f12c94f2aa2cec5883aa"
server_key_version: "encryptionkey@0"
I20260925 00:48:37.070091 4907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "3f7764946e4a49b8b3f95afaf4742955"
format_stamp: "Formatted at 2026-09-25 00:48:37 on dist-test-slave-05m0"
server_key: "0f4c7ad05bd36cb117a20325e31ccbde"
server_key_iv: "e048cf742d76f12c94f2aa2cec5883aa"
server_key_version: "encryptionkey@0"
I20260925 00:48:37.073675 4907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 00:48:37.075922 4922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:37.077071 4907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:37.077200 4907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "3f7764946e4a49b8b3f95afaf4742955"
format_stamp: "Formatted at 2026-09-25 00:48:37 on dist-test-slave-05m0"
server_key: "0f4c7ad05bd36cb117a20325e31ccbde"
server_key_iv: "e048cf742d76f12c94f2aa2cec5883aa"
server_key_version: "encryptionkey@0"
I20260925 00:48:37.077304 4907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:37.091805 4907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:37.094988 4907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:37.095192 4907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:37.095731 4907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:37.096496 4907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:37.096571 4907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:37.096637 4907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:37.096685 4907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:37.105223 4907 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:41827
I20260925 00:48:37.105263 5035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:41827 every 8 connection(s)
I20260925 00:48:37.106127 4907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
Sep 25 00:48:37 dist-test-slave-05m0 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:37.115782 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 4907
I20260925 00:48:37.115927 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:37.116293 997 external_mini_cluster.cc:1428] Setting key 256650fa71f9469b3d88290fc936e1f4
I20260925 00:48:37.119971 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:37.107791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:57755 (local address 127.0.249.126:38303)
0925 00:48:37.107921 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:37.107925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:37.108659 (+ 734us) server_negotiation.cc:407] Connection header received
0925 00:48:37.110051 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:37.110054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:37.110113 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:37.110215 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:37.111315 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:37.111836 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:37.112932 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:37.113159 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:37.113267 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:37.115572 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:37.115577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:37.115583 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:37.115617 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:37.117447 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:37.118090 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:37.118098 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:37.118100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:37.118219 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:37.118657 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:37.118662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:37.118665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:37.118902 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:37.119036 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:37.119631 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 00:48:37.119776 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:48:37.120807 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:37.108075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:38303 (local address 127.0.249.67:57755)
0925 00:48:37.108474 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:37.108513 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:37.109831 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:37.110377 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:37.110384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:37.110734 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:37.111168 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:37.111180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:37.111955 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:37.111959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:37.112805 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:37.112814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:37.113302 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:37.113306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:37.113495 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:37.113993 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:37.114008 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:37.115396 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:37.117582 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:37.117589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:37.117604 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:37.117958 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:37.118337 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:37.118346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:37.118349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:37.118530 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:37.119031 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:37.119038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:37.119291 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 00:48:37.119514 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 00:48:37.119908 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260925 00:48:37.121968 5036 heartbeater.cc:347] Connected to a master server at 127.0.249.126:38303
I20260925 00:48:37.122210 5036 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:37.122655 5036 heartbeater.cc:511] Master 127.0.249.126:38303 requested a full tablet report, sending...
I20260925 00:48:37.123677 4577 ts_manager.cc:194] Registered new tserver with Master: 3f7764946e4a49b8b3f95afaf4742955 (127.0.249.67:41827)
I20260925 00:48:37.124402 4577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:57755
I20260925 00:48:37.130038 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 00:48:37 dist-test-slave-05m0 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 00:48:37 dist-test-slave-05m0 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:37.155006 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:37.145987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.0.249.65:44771)
0925 00:48:37.146314 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:37.146328 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:37.146352 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:48:37.146400 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:37.146405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:37.146582 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:37.146698 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:37.147313 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:37.148061 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:37.148954 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:37.149339 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:37.149501 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:37.151741 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:37.151750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:37.151761 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:37.151784 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:37.153191 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:37.153678 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:37.153684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:37.153685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:37.153761 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:37.154051 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:37.154054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:37.154056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:37.154284 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:37.154530 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:37.154690 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 00:48:37.154848 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":154,"threads_started":1}
Sep 25 00:48:37 dist-test-slave-05m0 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:37.165130 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:37.157692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.249.126:38303)
0925 00:48:37.157804 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:37.157808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:37.157824 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:48:37.157929 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:37.157931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:37.157968 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:37.158024 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:37.158694 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:37.159184 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:37.159890 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:37.160090 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:37.160162 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:37.162357 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:37.162360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:37.162362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:37.162384 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:37.163652 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:37.163997 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:37.164000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:37.164001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:37.164073 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:37.164355 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:37.164358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:37.164359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:37.164591 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:37.164762 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:37.164923 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 00:48:37.165019 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 00:48:37.165560 4577 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43970
I20260925 00:48:37.166111 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 4639
I20260925 00:48:37.171619 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 4773
I20260925 00:48:37.176693 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 4907
I20260925 00:48:37.181752 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 4545
2026-09-25T00:48:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2100 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:37 dist-test-slave-05m0 krb5kdc[5048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:37 dist-test-slave-05m0 krb5kdc[5048](info): set up 2 sockets
Sep 25 00:48:37 dist-test-slave-05m0 krb5kdc[5048](info): commencing operation
krb5kdc: starting...
W20260925 00:48:38.221562 997 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:38 dist-test-slave-05m0 krb5kdc[5048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:38.343833 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:42731
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:42731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:38.450757 5061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:38.451028 5061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:38.451071 5061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:38.454999 5061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:38.455055 5061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:38.455075 5061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:38.455090 5061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:38.455106 5061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:38.460312 5061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:42731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:38.461314 5061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:38.462473 5061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:38.470705 5069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:38.470726 5067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:38.470705 5066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:38.471086 5061 server_base.cc:1034] running on GCE node
I20260925 00:48:38.471628 5061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:38.472711 5061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:38.473866 5061 hybrid_clock.cc:648] HybridClock initialized: now 1790297318473834 us; error 46 us; skew 500 ppm
Sep 25 00:48:38 dist-test-slave-05m0 krb5kdc[5048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:38.480088 5061 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:38.480854 5061 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:38.480952 5061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:38.481197 5061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:38.483321 5061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "aa11b3cfd272415ab78be8a0ee784db9"
format_stamp: "Formatted at 2026-09-25 00:48:38 on dist-test-slave-05m0"
server_key: "754c2bfb0db0c44dae1c3dabcde7b594"
server_key_iv: "e9fee16cdcea33132e4cc4a745b7f447"
server_key_version: "encryptionkey@0"
I20260925 00:48:38.483904 5061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "aa11b3cfd272415ab78be8a0ee784db9"
format_stamp: "Formatted at 2026-09-25 00:48:38 on dist-test-slave-05m0"
server_key: "754c2bfb0db0c44dae1c3dabcde7b594"
server_key_iv: "e9fee16cdcea33132e4cc4a745b7f447"
server_key_version: "encryptionkey@0"
I20260925 00:48:38.487694 5061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:48:38.490274 5075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:38.491631 5061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:38.491792 5061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "aa11b3cfd272415ab78be8a0ee784db9"
format_stamp: "Formatted at 2026-09-25 00:48:38 on dist-test-slave-05m0"
server_key: "754c2bfb0db0c44dae1c3dabcde7b594"
server_key_iv: "e9fee16cdcea33132e4cc4a745b7f447"
server_key_version: "encryptionkey@0"
I20260925 00:48:38.491919 5061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:38.503098 5061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:38.506354 5061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:38.506561 5061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:38.513630 5061 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:42731
I20260925 00:48:38.513906 5127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:42731 every 8 connection(s)
I20260925 00:48:38.514299 5061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:38.516921 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:38.519341 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 5061
I20260925 00:48:38.519531 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:38.519809 997 external_mini_cluster.cc:1428] Setting key 5f6601d1279aee6784361781e7cd9fbe
I20260925 00:48:38.522692 5128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9: Bootstrap starting.
I20260925 00:48:38.525278 5128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:38.526196 5128 log.cc:824] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:38.528613 5128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9: No bootstrap required, opened a new log
Sep 25 00:48:38 dist-test-slave-05m0 krb5kdc[5048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:38.532824 5128 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa11b3cfd272415ab78be8a0ee784db9" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42731 } }
I20260925 00:48:38.533077 5128 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:38.533131 5128 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa11b3cfd272415ab78be8a0ee784db9, State: Initialized, Role: FOLLOWER
I20260925 00:48:38.533627 5128 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa11b3cfd272415ab78be8a0ee784db9" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42731 } }
I20260925 00:48:38.533793 5128 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:38.533874 5128 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:38.533999 5128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:38.534941 5128 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa11b3cfd272415ab78be8a0ee784db9" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42731 } }
I20260925 00:48:38.535331 5128 leader_election.cc:302] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa11b3cfd272415ab78be8a0ee784db9; no voters:
I20260925 00:48:38.535701 5128 leader_election.cc:288] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:38.535773 5133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:38.535976 5133 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [term 1 LEADER]: Becoming Leader. State: Replica: aa11b3cfd272415ab78be8a0ee784db9, State: Running, Role: LEADER
I20260925 00:48:38.536124 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:38.521254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.0.249.126:42731)
0925 00:48:38.521685 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:38.521712 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:38.521747 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:48:38.522362 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:38.522379 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:38.522721 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:38.523051 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:38.523929 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:38.524927 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:38.526148 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:38.526795 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:38.526967 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:38.529905 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:38.529912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:38.529929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:38.529961 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:38.532621 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:38.533160 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:38.533187 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:38.533195 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:38.533369 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:38.533699 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:38.533703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:38.533705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:38.534265 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:38.534500 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:38.534784 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 00:48:38.535132 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":177,"threads_started":1}
I20260925 00:48:38.536295 5133 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa11b3cfd272415ab78be8a0ee784db9" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42731 } }
I20260925 00:48:38.537031 5128 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:38.538934 5135 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa11b3cfd272415ab78be8a0ee784db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa11b3cfd272415ab78be8a0ee784db9" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42731 } } }
I20260925 00:48:38.539350 5135 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:38.539580 5134 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa11b3cfd272415ab78be8a0ee784db9. Latest consensus state: current_term: 1 leader_uuid: "aa11b3cfd272415ab78be8a0ee784db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa11b3cfd272415ab78be8a0ee784db9" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42731 } } }
I20260925 00:48:38.539861 5134 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9 [sys.catalog]: This master's current role is: LEADER
W20260925 00:48:38.542054 5148 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:48:38.542147 5148 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:48:38.542230 5149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:38.542861 5149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:38.548382 5149 catalog_manager.cc:1347] Generated new cluster ID: 7905e93b17b44018821fc1f4eef0e991
I20260925 00:48:38.548451 5149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:38.556056 5149 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:38.557231 5149 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:38.566248 5149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9: Generated new TSK 0
I20260925 00:48:38.566927 5149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:48:38.601166 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:42731
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:38.707540 5154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:38.707814 5154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:38.707892 5154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:38.711895 5154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:38.711978 5154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:38.712073 5154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:38.717072 5154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:38.718127 5154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:38.719115 5154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:38.728348 5160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:38.728348 5159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:38.728361 5162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:38.728730 5154 server_base.cc:1034] running on GCE node
I20260925 00:48:38.729147 5154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:38.729830 5154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:38.730999 5154 hybrid_clock.cc:648] HybridClock initialized: now 1790297318730990 us; error 31 us; skew 500 ppm
Sep 25 00:48:38 dist-test-slave-05m0 krb5kdc[5048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:38.733892 5154 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:38.734659 5154 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:38.734758 5154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:38.735044 5154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:38.737339 5154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "28dac1f7a4cd4512b9e8754478533975"
format_stamp: "Formatted at 2026-09-25 00:48:38 on dist-test-slave-05m0"
server_key: "c4f64dd72f8a5fafc7f6549e7805d738"
server_key_iv: "62d43b01d4e1ee0cc46ab8c770838afd"
server_key_version: "encryptionkey@0"
I20260925 00:48:38.738144 5154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "28dac1f7a4cd4512b9e8754478533975"
format_stamp: "Formatted at 2026-09-25 00:48:38 on dist-test-slave-05m0"
server_key: "c4f64dd72f8a5fafc7f6549e7805d738"
server_key_iv: "62d43b01d4e1ee0cc46ab8c770838afd"
server_key_version: "encryptionkey@0"
I20260925 00:48:38.742199 5154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 00:48:38.744642 5168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:38.745915 5154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 00:48:38.746052 5154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "28dac1f7a4cd4512b9e8754478533975"
format_stamp: "Formatted at 2026-09-25 00:48:38 on dist-test-slave-05m0"
server_key: "c4f64dd72f8a5fafc7f6549e7805d738"
server_key_iv: "62d43b01d4e1ee0cc46ab8c770838afd"
server_key_version: "encryptionkey@0"
I20260925 00:48:38.746168 5154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:38.773044 5154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:38.776072 5154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:38.776275 5154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:38.776877 5154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:38.777833 5154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:38.777930 5154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:38.778031 5154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:38.778086 5154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:38.788002 5154 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:43825
I20260925 00:48:38.788023 5281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:43825 every 8 connection(s)
I20260925 00:48:38.788599 5154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:38.797771 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 5154
I20260925 00:48:38.797915 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:38.798192 997 external_mini_cluster.cc:1428] Setting key eedc67fd05a07585eddc7eb4522ffd12
Sep 25 00:48:38 dist-test-slave-05m0 krb5kdc[5048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:38.802423 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:38.790434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:46869 (local address 127.0.249.126:42731)
0925 00:48:38.790596 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:38.790600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:38.791423 (+ 823us) server_negotiation.cc:407] Connection header received
0925 00:48:38.792763 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:38.792766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:38.792824 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:38.792932 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:38.794089 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:38.794675 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:38.795872 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:38.796120 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:38.796230 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:38.798764 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:38.798768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:38.798771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:38.798804 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:38.800366 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:38.800870 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:38.800875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:38.800876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:38.800961 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:38.801305 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:38.801309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:38.801310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:38.801553 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:38.801658 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:38.802149 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 00:48:38.802265 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:48:38.803231 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:38.790836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:42731 (local address 127.0.249.65:46869)
0925 00:48:38.791279 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:38.791319 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:38.792552 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:38.793113 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:38.793120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:38.793475 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:38.793927 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:38.793940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:38.794806 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:38.794814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:38.795743 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:38.795752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:38.796267 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:38.796272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:38.796441 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:38.797045 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:38.797061 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:38.798602 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:38.800491 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:38.800496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:38.800511 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:38.800766 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:38.801058 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:38.801061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:38.801063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:38.801202 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:38.801657 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:38.801663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:38.801910 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 00:48:38.802109 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 00:48:38.802423 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260925 00:48:38.804598 5282 heartbeater.cc:347] Connected to a master server at 127.0.249.126:42731
I20260925 00:48:38.804839 5282 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:38.805277 5282 heartbeater.cc:511] Master 127.0.249.126:42731 requested a full tablet report, sending...
I20260925 00:48:38.806797 5092 ts_manager.cc:194] Registered new tserver with Master: 28dac1f7a4cd4512b9e8754478533975 (127.0.249.65:43825)
I20260925 00:48:38.807883 5092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:46869
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:38.826442 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:42731
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:38.938026 5287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:38.938270 5287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:38.938313 5287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:38.942323 5287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:38.942373 5287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:38.942436 5287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:38.947436 5287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:38.948309 5287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:38.949333 5287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:38.958441 5292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:38.958557 5287 server_base.cc:1034] running on GCE node
W20260925 00:48:38.958494 5295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:38.958441 5293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:38.959065 5287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:38.959682 5287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:38.960861 5287 hybrid_clock.cc:648] HybridClock initialized: now 1790297318960831 us; error 41 us; skew 500 ppm
Sep 25 00:48:38 dist-test-slave-05m0 krb5kdc[5048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:38.964000 5287 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:38.964901 5287 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:38.964996 5287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:38.965209 5287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:38.967391 5287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "73e1133eeba240a5b0ab804ba01a9432"
format_stamp: "Formatted at 2026-09-25 00:48:38 on dist-test-slave-05m0"
server_key: "e125bdbdee3c5ff6cf18f1dff29d7125"
server_key_iv: "15e0ae577f11fcf3f049dc8b6736ff00"
server_key_version: "encryptionkey@0"
I20260925 00:48:38.967943 5287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "73e1133eeba240a5b0ab804ba01a9432"
format_stamp: "Formatted at 2026-09-25 00:48:38 on dist-test-slave-05m0"
server_key: "e125bdbdee3c5ff6cf18f1dff29d7125"
server_key_iv: "15e0ae577f11fcf3f049dc8b6736ff00"
server_key_version: "encryptionkey@0"
I20260925 00:48:38.971572 5287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 00:48:38.973845 5301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:38.974975 5287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:38.975117 5287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "73e1133eeba240a5b0ab804ba01a9432"
format_stamp: "Formatted at 2026-09-25 00:48:38 on dist-test-slave-05m0"
server_key: "e125bdbdee3c5ff6cf18f1dff29d7125"
server_key_iv: "15e0ae577f11fcf3f049dc8b6736ff00"
server_key_version: "encryptionkey@0"
I20260925 00:48:38.975220 5287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:38.992573 5287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:38.996210 5287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:38.996428 5287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:38.997030 5287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:38.998108 5287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:38.998188 5287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:38.998265 5287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:38.998322 5287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:39.007737 5287 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:37785
I20260925 00:48:39.007772 5414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:37785 every 8 connection(s)
I20260925 00:48:39.008334 5287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:39.012545 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 5287
I20260925 00:48:39.012681 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:39.012976 997 external_mini_cluster.cc:1428] Setting key cb0f9797c41675dce532dbf5d8b75b0f
Sep 25 00:48:39 dist-test-slave-05m0 krb5kdc[5048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:48:39.023929 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.010436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:50517 (local address 127.0.249.126:42731)
0925 00:48:39.010575 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.010579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.011405 (+ 826us) server_negotiation.cc:407] Connection header received
0925 00:48:39.012996 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.013000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.013057 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.013150 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:39.014612 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.015194 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.017021 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.017237 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.017321 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.020267 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:39.020276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:39.020278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:39.020300 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:39.021873 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:39.022402 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:39.022407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:39.022409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:39.022489 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:39.022815 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:39.022820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:39.022822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:39.023041 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:39.023145 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.023629 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.023750 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:48:39.024717 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.010782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:42731 (local address 127.0.249.66:50517)
0925 00:48:39.011249 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.011296 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:39.012697 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:39.013366 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:39.013376 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:39.013829 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:39.014463 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.014481 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.015329 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.015333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.016874 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.016888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.017426 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.017434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.017698 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.018353 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:39.018381 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:39.020104 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:39.022015 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:39.022021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:39.022035 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:39.022296 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:39.022582 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:39.022585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:39.022587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:39.022720 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:39.023143 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:39.023150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:39.023385 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 00:48:39.023580 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 00:48:39.023890 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
I20260925 00:48:39.025955 5415 heartbeater.cc:347] Connected to a master server at 127.0.249.126:42731
I20260925 00:48:39.026204 5415 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:39.026688 5415 heartbeater.cc:511] Master 127.0.249.126:42731 requested a full tablet report, sending...
I20260925 00:48:39.027762 5092 ts_manager.cc:194] Registered new tserver with Master: 73e1133eeba240a5b0ab804ba01a9432 (127.0.249.66:37785)
I20260925 00:48:39.028800 5092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:50517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:39.040448 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:42731
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:39.155201 5420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:39.155484 5420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:39.155573 5420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:39.159593 5420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:39.159686 5420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:39.159783 5420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:39.164996 5420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:39.166043 5420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:39.167042 5420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:39.176332 5426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:39.176292 5428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:39.176586 5420 server_base.cc:1034] running on GCE node
W20260925 00:48:39.176663 5425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:39.177044 5420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:39.177695 5420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:39.178973 5420 hybrid_clock.cc:648] HybridClock initialized: now 1790297319178946 us; error 35 us; skew 500 ppm
Sep 25 00:48:39 dist-test-slave-05m0 krb5kdc[5048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:39.181907 5420 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:39.182633 5420 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:39.182723 5420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:39.182932 5420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:39.184906 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "f660728d01334941aba33b207ded3173"
format_stamp: "Formatted at 2026-09-25 00:48:39 on dist-test-slave-05m0"
server_key: "12306d927cedd30f921ef901620a3591"
server_key_iv: "b6c2d992343e17d8f885165dd70c473b"
server_key_version: "encryptionkey@0"
I20260925 00:48:39.185518 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "f660728d01334941aba33b207ded3173"
format_stamp: "Formatted at 2026-09-25 00:48:39 on dist-test-slave-05m0"
server_key: "12306d927cedd30f921ef901620a3591"
server_key_iv: "b6c2d992343e17d8f885165dd70c473b"
server_key_version: "encryptionkey@0"
I20260925 00:48:39.188912 5420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:48:39.191291 5434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:39.192389 5420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:39.192529 5420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "f660728d01334941aba33b207ded3173"
format_stamp: "Formatted at 2026-09-25 00:48:39 on dist-test-slave-05m0"
server_key: "12306d927cedd30f921ef901620a3591"
server_key_iv: "b6c2d992343e17d8f885165dd70c473b"
server_key_version: "encryptionkey@0"
I20260925 00:48:39.192637 5420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:39.215529 5420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:39.218890 5420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:39.219094 5420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:39.219669 5420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:39.220460 5420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:39.220536 5420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:39.220625 5420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:39.220675 5420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:39.229542 5420 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:33797
I20260925 00:48:39.229574 5547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:33797 every 8 connection(s)
I20260925 00:48:39.230156 5420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:39.236316 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 5420
I20260925 00:48:39.236445 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:39.236768 997 external_mini_cluster.cc:1428] Setting key 381a47b856c7f925b834d32b48201fbb
Sep 25 00:48:39 dist-test-slave-05m0 krb5kdc[5048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:39.243741 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.231845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:36431 (local address 127.0.249.126:42731)
0925 00:48:39.231979 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.231982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.232776 (+ 794us) server_negotiation.cc:407] Connection header received
0925 00:48:39.234195 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.234198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.234257 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.234332 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:39.235440 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.235972 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.237416 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.237719 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.237832 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.240430 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:39.240433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:39.240436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:39.240464 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:39.241778 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:39.242280 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:39.242283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:39.242285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:39.242364 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:39.242703 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:39.242708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:39.242710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:39.242891 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:39.242996 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.243464 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.243584 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:48:39.244544 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.232218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:42731 (local address 127.0.249.67:36431)
0925 00:48:39.232638 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.232680 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:39.233972 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:39.234495 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:39.234502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:39.234842 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:39.235298 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.235310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.236112 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.236115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.237274 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.237292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.237863 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.237866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.238104 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.238750 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:39.238775 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:39.240268 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:39.241889 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:39.241895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:39.241910 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:39.242167 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:39.242458 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:39.242461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:39.242463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:39.242607 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:39.242993 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:39.242999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:39.243240 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 00:48:39.243424 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 00:48:39.243738 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":113,"threads_started":1}
I20260925 00:48:39.245739 5548 heartbeater.cc:347] Connected to a master server at 127.0.249.126:42731
I20260925 00:48:39.246016 5548 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:39.246460 5548 heartbeater.cc:511] Master 127.0.249.126:42731 requested a full tablet report, sending...
I20260925 00:48:39.247496 5092 ts_manager.cc:194] Registered new tserver with Master: f660728d01334941aba33b207ded3173 (127.0.249.67:33797)
I20260925 00:48:39.248070 5092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:36431
I20260925 00:48:39.250666 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:48:39.259794 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.252622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.0.249.126:42731)
0925 00:48:39.252731 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.252755 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.252954 (+ 199us) server_negotiation.cc:407] Connection header received
0925 00:48:39.253099 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.253102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.253146 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.253225 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:39.254053 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.254542 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.255368 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.255589 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.255694 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.256929 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:39.256933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:39.256936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:39.256959 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:39.258174 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:39.258586 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:39.258591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:39.258593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:39.258671 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:39.258945 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:39.258948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:39.258950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:39.259158 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:39.259296 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.259530 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.259638 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:48:39.262394 5092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:39.265491 5092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:39.279295 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.273360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.0.249.66:37785)
0925 00:48:39.274238 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.274250 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.274274 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:48:39.274351 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.274357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.274574 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.274730 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:39.275229 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.276116 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.277761 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.278607 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.278726 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.278949 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.279012 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.279107 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":120,"threads_started":1}
I20260925 00:48:39.279295 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.273241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:37785 (local address 127.0.0.1:52514)
0925 00:48:39.273765 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.273810 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:39.273941 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:39.274947 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:39.274960 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:39.274979 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:39.275083 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.275088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.276237 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.276243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.277614 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.277624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.278725 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.278728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.278845 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.278886 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 00:48:39.278978 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 00:48:39.279097 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":47,"spinlock_wait_cycles":1792,"thread_start_us":81,"threads_started":1}
I20260925 00:48:39.279309 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.273354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:33797 (local address 127.0.0.1:46834)
0925 00:48:39.273765 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.273810 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:39.273940 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:39.275177 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:39.275181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:39.275200 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:39.275285 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.275290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.276690 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.276693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.277845 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.277852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.278845 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.278851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.278956 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.278981 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:48:39.279035 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 00:48:39.279115 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":29,"spinlock_wait_cycles":6016,"thread_start_us":118,"threads_started":1}
I20260925 00:48:39.279700 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.273991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:43825 (local address 127.0.0.1:41844)
0925 00:48:39.274535 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.274550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:39.274635 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:39.275448 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:39.275451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:39.275467 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:39.275539 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.275545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.276808 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.276813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.278148 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.278159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.279259 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.279262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.279362 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.279391 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 00:48:39.279456 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 00:48:39.279540 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":162,"thread_start_us":84,"threads_started":1}
I20260925 00:48:39.280273 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.274065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.0.249.67:33797)
0925 00:48:39.274746 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.274762 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.274782 (+ 20us) server_negotiation.cc:407] Connection header received
0925 00:48:39.274852 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.274858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.275039 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.275185 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:39.275403 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.276548 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.277954 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.278707 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.279861 (+ 1154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.279995 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.280046 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.280125 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":141,"threads_started":1}
I20260925 00:48:39.280273 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.274306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.0.249.65:43825)
0925 00:48:39.274981 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.274987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.275005 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:48:39.275087 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.275095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.275267 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.275486 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:39.275636 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.276686 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.278268 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.279071 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.279865 (+ 794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.279995 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.280046 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.280126 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":76,"threads_started":1}
I20260925 00:48:39.282366 5349 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5b610085354200a11467d299a713e2 (DEFAULT_TABLE table=test-table [id=6e2306a4c2594674b469f339299c33cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:39.282982 5216 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5b610085354200a11467d299a713e2 (DEFAULT_TABLE table=test-table [id=6e2306a4c2594674b469f339299c33cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:39.283126 5482 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5b610085354200a11467d299a713e2 (DEFAULT_TABLE table=test-table [id=6e2306a4c2594674b469f339299c33cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:39.283466 5349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5b610085354200a11467d299a713e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:39.283959 5482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5b610085354200a11467d299a713e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:39.283962 5216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5b610085354200a11467d299a713e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:39.289765 5565 tablet_bootstrap.cc:492] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: Bootstrap starting.
I20260925 00:48:39.290967 5566 tablet_bootstrap.cc:492] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975: Bootstrap starting.
I20260925 00:48:39.292161 5565 tablet_bootstrap.cc:654] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:39.293072 5566 tablet_bootstrap.cc:654] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:39.293586 5565 log.cc:824] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:39.293713 5567 tablet_bootstrap.cc:492] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: Bootstrap starting.
I20260925 00:48:39.294381 5566 log.cc:824] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:39.295606 5565 tablet_bootstrap.cc:492] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: No bootstrap required, opened a new log
I20260925 00:48:39.295730 5567 tablet_bootstrap.cc:654] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:39.295787 5565 ts_tablet_manager.cc:1397] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260925 00:48:39.296353 5566 tablet_bootstrap.cc:492] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975: No bootstrap required, opened a new log
I20260925 00:48:39.296533 5566 ts_tablet_manager.cc:1397] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 00:48:39.296988 5567 log.cc:824] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:39.298511 5567 tablet_bootstrap.cc:492] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: No bootstrap required, opened a new log
I20260925 00:48:39.298710 5567 ts_tablet_manager.cc:1397] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:48:39.299649 5565 raft_consensus.cc:348] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.299938 5565 raft_consensus.cc:374] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:39.300004 5565 raft_consensus.cc:729] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f660728d01334941aba33b207ded3173, State: Initialized, Role: FOLLOWER
I20260925 00:48:39.300215 5566 raft_consensus.cc:348] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.300454 5566 raft_consensus.cc:374] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:39.300504 5566 raft_consensus.cc:729] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dac1f7a4cd4512b9e8754478533975, State: Initialized, Role: FOLLOWER
I20260925 00:48:39.300436 5565 consensus_queue.cc:260] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.300918 5566 consensus_queue.cc:260] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.301430 5565 ts_tablet_manager.cc:1428] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260925 00:48:39.301582 5282 heartbeater.cc:503] Master 127.0.249.126:42731 was elected leader, sending a full tablet report...
I20260925 00:48:39.301745 5548 heartbeater.cc:503] Master 127.0.249.126:42731 was elected leader, sending a full tablet report...
I20260925 00:48:39.301858 5566 ts_tablet_manager.cc:1428] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 00:48:39.302245 5567 raft_consensus.cc:348] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.302489 5567 raft_consensus.cc:374] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:39.302567 5567 raft_consensus.cc:729] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e1133eeba240a5b0ab804ba01a9432, State: Initialized, Role: FOLLOWER
I20260925 00:48:39.302994 5567 consensus_queue.cc:260] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.303890 5567 ts_tablet_manager.cc:1428] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:48:39.304212 5415 heartbeater.cc:503] Master 127.0.249.126:42731 was elected leader, sending a full tablet report...
W20260925 00:48:39.480991 5549 tablet.cc:2369] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:48:39.509451 5416 tablet.cc:2369] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:48:39.539882 5283 tablet.cc:2369] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:39.592999 5573 raft_consensus.cc:482] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:39.593200 5573 raft_consensus.cc:504] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.594440 5573 leader_election.cc:288] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f660728d01334941aba33b207ded3173 (127.0.249.67:33797), 28dac1f7a4cd4512b9e8754478533975 (127.0.249.65:43825)
I20260925 00:48:39.595180 5571 raft_consensus.cc:482] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:39.595386 5571 raft_consensus.cc:504] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.596585 5571 leader_election.cc:288] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73e1133eeba240a5b0ab804ba01a9432 (127.0.249.66:37785), 28dac1f7a4cd4512b9e8754478533975 (127.0.249.65:43825)
I20260925 00:48:39.599381 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.594851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:37769 (local address 127.0.249.65:43825)
0925 00:48:39.594960 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.594965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.595135 (+ 170us) server_negotiation.cc:407] Connection header received
0925 00:48:39.595318 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.595323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.595386 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.595457 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:39.595824 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.596669 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.598195 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.598782 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.598905 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.598963 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.599198 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.599262 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:48:39.599413 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.594703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:43825 (local address 127.0.249.66:37769)
0925 00:48:39.595017 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.595042 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:39.595170 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:39.595488 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:39.595491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:39.595510 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:39.595713 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.595721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.596879 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.596886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.598070 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.598078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.598960 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.598963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.599066 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.599083 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:48:39.599182 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 00:48:39.599249 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":45,"spinlock_wait_cycles":7040,"thread_start_us":91,"threads_started":1}
I20260925 00:48:39.599713 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.594728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:48723 (local address 127.0.249.67:33797)
0925 00:48:39.594861 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.594866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.595042 (+ 176us) server_negotiation.cc:407] Connection header received
0925 00:48:39.595275 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.595279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.595330 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.595380 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:39.595958 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.596668 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.598326 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.598940 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.599044 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.599088 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.599524 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.599601 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":51}
I20260925 00:48:39.600075 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.594619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:33797 (local address 127.0.249.66:48723)
0925 00:48:39.594959 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.594974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:39.595090 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:39.595529 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:39.595535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:39.595566 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:39.595793 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.595803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.596796 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.596799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.598194 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.598203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.599150 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.599153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.599253 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.599272 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 00:48:39.599871 (+ 599us) client_negotiation.cc:240] Negotiation successful
0925 00:48:39.599935 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":94,"thread_start_us":87,"threads_started":1}
I20260925 00:48:39.600653 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5b610085354200a11467d299a713e2" candidate_uuid: "73e1133eeba240a5b0ab804ba01a9432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f660728d01334941aba33b207ded3173" is_pre_election: true
I20260925 00:48:39.600780 5236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5b610085354200a11467d299a713e2" candidate_uuid: "73e1133eeba240a5b0ab804ba01a9432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dac1f7a4cd4512b9e8754478533975" is_pre_election: true
I20260925 00:48:39.600855 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.597024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:40399 (local address 127.0.249.66:37785)
0925 00:48:39.597122 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.597126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.597137 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:48:39.597172 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.597175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.597209 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.597277 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:39.597618 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.598281 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.599609 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.600138 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.600237 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.600276 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.600700 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.600760 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 00:48:39.600878 5502 raft_consensus.cc:2454] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73e1133eeba240a5b0ab804ba01a9432 in term 0.
I20260925 00:48:39.600616 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.596724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:37785 (local address 127.0.249.67:40399)
0925 00:48:39.596819 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.596833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:39.596893 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:39.597364 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:39.597366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:39.597382 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:39.597491 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.597497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.598384 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.598387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.599480 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.599489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.600232 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.600235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.600339 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.600370 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 00:48:39.600439 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 00:48:39.600500 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 00:48:39.601047 5236 raft_consensus.cc:2454] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73e1133eeba240a5b0ab804ba01a9432 in term 0.
I20260925 00:48:39.601436 5303 leader_election.cc:302] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73e1133eeba240a5b0ab804ba01a9432, f660728d01334941aba33b207ded3173; no voters:
I20260925 00:48:39.601644 5369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5b610085354200a11467d299a713e2" candidate_uuid: "f660728d01334941aba33b207ded3173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e1133eeba240a5b0ab804ba01a9432" is_pre_election: true
I20260925 00:48:39.601699 5573 raft_consensus.cc:2781] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:39.601819 5369 raft_consensus.cc:2454] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f660728d01334941aba33b207ded3173 in term 0.
I20260925 00:48:39.601900 5573 raft_consensus.cc:482] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:39.601969 5573 raft_consensus.cc:3037] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:39.602948 5573 raft_consensus.cc:504] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.603085 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.597339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:53349 (local address 127.0.249.65:43825)
0925 00:48:39.597676 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.597680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.598535 (+ 855us) server_negotiation.cc:407] Connection header received
0925 00:48:39.598586 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.598589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.598627 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.598693 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:39.599718 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.600361 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.602076 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.602552 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.602677 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.602718 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.602902 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.602968 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":118,"threads_started":1}
I20260925 00:48:39.603088 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.596982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:43825 (local address 127.0.249.67:53349)
0925 00:48:39.598368 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.598382 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:39.598465 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:39.598774 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:39.598776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:39.598804 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:39.599396 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.599405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.600481 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.600484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.601951 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:39.601961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.602680 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.602683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:39.602785 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.602801 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 00:48:39.602888 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 00:48:39.602948 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"mutex_wait_us":440,"thread_start_us":98,"threads_started":1}
I20260925 00:48:39.603331 5573 leader_election.cc:288] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [CANDIDATE]: Term 1 election: Requested vote from peers f660728d01334941aba33b207ded3173 (127.0.249.67:33797), 28dac1f7a4cd4512b9e8754478533975 (127.0.249.65:43825)
I20260925 00:48:39.603616 5236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5b610085354200a11467d299a713e2" candidate_uuid: "f660728d01334941aba33b207ded3173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dac1f7a4cd4512b9e8754478533975" is_pre_election: true
I20260925 00:48:39.603729 5236 raft_consensus.cc:2454] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f660728d01334941aba33b207ded3173 in term 0.
I20260925 00:48:39.603727 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5b610085354200a11467d299a713e2" candidate_uuid: "73e1133eeba240a5b0ab804ba01a9432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f660728d01334941aba33b207ded3173"
I20260925 00:48:39.603821 5502 raft_consensus.cc:3037] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:39.603981 5435 leader_election.cc:302] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73e1133eeba240a5b0ab804ba01a9432, f660728d01334941aba33b207ded3173; no voters:
I20260925 00:48:39.604171 5236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5b610085354200a11467d299a713e2" candidate_uuid: "73e1133eeba240a5b0ab804ba01a9432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dac1f7a4cd4512b9e8754478533975"
I20260925 00:48:39.604267 5236 raft_consensus.cc:3037] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:39.605604 5502 raft_consensus.cc:2454] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73e1133eeba240a5b0ab804ba01a9432 in term 1.
I20260925 00:48:39.605615 5236 raft_consensus.cc:2454] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73e1133eeba240a5b0ab804ba01a9432 in term 1.
I20260925 00:48:39.605731 5571 raft_consensus.cc:2741] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 00:48:39.605967 5302 leader_election.cc:302] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dac1f7a4cd4512b9e8754478533975, 73e1133eeba240a5b0ab804ba01a9432; no voters:
I20260925 00:48:39.606177 5573 raft_consensus.cc:2781] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:39.606381 5573 raft_consensus.cc:686] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 1 LEADER]: Becoming Leader. State: Replica: 73e1133eeba240a5b0ab804ba01a9432, State: Running, Role: LEADER
I20260925 00:48:39.606715 5573 consensus_queue.cc:237] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } }
I20260925 00:48:39.609726 5090 catalog_manager.cc:5495] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73e1133eeba240a5b0ab804ba01a9432 (127.0.249.66). New cstate: current_term: 1 leader_uuid: "73e1133eeba240a5b0ab804ba01a9432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e1133eeba240a5b0ab804ba01a9432" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 } health_report { overall_health: UNKNOWN } } }
I20260925 00:48:39.647379 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.643784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.0.249.66:37785)
0925 00:48:39.643967 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.643973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.643990 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:48:39.644074 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.644076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.644125 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.644239 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:39.644497 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.645013 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.645925 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:39.646140 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:39.646254 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:39.646576 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:39.647002 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:39.647139 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.647180 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.647231 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 00:48:39.653424 5502 raft_consensus.cc:1261] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 1 FOLLOWER]: Refusing update from remote peer 73e1133eeba240a5b0ab804ba01a9432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:39.653476 5236 raft_consensus.cc:1261] T 9e5b610085354200a11467d299a713e2 P 28dac1f7a4cd4512b9e8754478533975 [term 1 FOLLOWER]: Refusing update from remote peer 73e1133eeba240a5b0ab804ba01a9432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:39.654124 5573 consensus_queue.cc:1035] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f660728d01334941aba33b207ded3173" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:39.654320 5578 consensus_queue.cc:1035] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dac1f7a4cd4512b9e8754478533975" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:39.661927 5583 mvcc.cc:205] Tried to move back new op lower bound from 7333057821293113344 to 7333057821111808000. Current Snapshot: MvccSnapshot[applied={T|T < 7333057821293113344}]
I20260925 00:48:39.663452 5582 mvcc.cc:205] Tried to move back new op lower bound from 7333057821293113344 to 7333057821111808000. Current Snapshot: MvccSnapshot[applied={T|T < 7333057821293113344}]
I20260925 00:48:39.670226 5090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:39.670406 5090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:39.672683 5090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa11b3cfd272415ab78be8a0ee784db9: Sending DeleteTablet for 3 replicas of tablet 9e5b610085354200a11467d299a713e2
I20260925 00:48:39.673521 5349 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5b610085354200a11467d299a713e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:39 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:52514
I20260925 00:48:39.673540 5482 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5b610085354200a11467d299a713e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:39 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:46834
I20260925 00:48:39.673717 5216 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5b610085354200a11467d299a713e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:39 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:41844
I20260925 00:48:39.674168 5591 tablet_replica.cc:330] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: stopping tablet replica
I20260925 00:48:39.674168 5590 tablet_replica.cc:330] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: stopping tablet replica
I20260925 00:48:39.674479 5590 raft_consensus.cc:2229] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:39.674479 5591 raft_consensus.cc:2229] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:39.674556 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 5154
I20260925 00:48:39.675166 5591 raft_consensus.cc:2258] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:39.676007 5590 raft_consensus.cc:2258] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:39.677527 5591 ts_tablet_manager.cc:1905] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:39.678288 5590 ts_tablet_manager.cc:1905] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:39.680891 5590 ts_tablet_manager.cc:1918] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:39.681000 5590 log.cc:1196] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/9e5b610085354200a11467d299a713e2
I20260925 00:48:39.681310 5590 ts_tablet_manager.cc:1939] T 9e5b610085354200a11467d299a713e2 P 73e1133eeba240a5b0ab804ba01a9432: Deleting consensus metadata
I20260925 00:48:39.681479 5591 ts_tablet_manager.cc:1918] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:39.681640 5591 log.cc:1196] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/9e5b610085354200a11467d299a713e2
I20260925 00:48:39.682047 5591 ts_tablet_manager.cc:1939] T 9e5b610085354200a11467d299a713e2 P f660728d01334941aba33b207ded3173: Deleting consensus metadata
I20260925 00:48:39.682569 5076 catalog_manager.cc:4841] TS 73e1133eeba240a5b0ab804ba01a9432 (127.0.249.66:37785): tablet 9e5b610085354200a11467d299a713e2 (table test-table [id=6e2306a4c2594674b469f339299c33cc]) successfully deleted
W20260925 00:48:39.682852 5076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:43825 (error 108)
I20260925 00:48:39.683090 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 5287
I20260925 00:48:39.684442 5077 catalog_manager.cc:4841] TS f660728d01334941aba33b207ded3173 (127.0.249.67:33797): tablet 9e5b610085354200a11467d299a713e2 (table test-table [id=6e2306a4c2594674b469f339299c33cc]) successfully deleted
I20260925 00:48:39.684815 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.684429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:43825 (local address 127.0.0.1:41846)
0925 00:48:39.684642 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:39.684712 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260925 00:48:39.684983 5076 catalog_manager.cc:4584] TS 28dac1f7a4cd4512b9e8754478533975 (127.0.249.65:43825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e5b610085354200a11467d299a713e2: Network error: Client connection negotiation failed: client connection to 127.0.249.65:43825: connect: Connection refused (error 111)
I20260925 00:48:39.688306 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 5420
I20260925 00:48:39.693637 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 5061
2026-09-25T00:48:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2512 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T00:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:39Z Disabled control of system clock
I20260925 00:48:39.722831 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:33771
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:34123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:33771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 00:48:39.829795 5597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:39.830062 5597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:39.830101 5597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:39.834116 5597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:39.834170 5597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:39.834187 5597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:39.834203 5597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:39.834218 5597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:39.838878 5597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:33771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:33771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:39.839833 5597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:39.841109 5597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:39.849191 5602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:39.849179 5605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:39.849313 5597 server_base.cc:1034] running on GCE node
W20260925 00:48:39.849176 5603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:39.849920 5597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:39.850936 5597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:39.852092 5597 hybrid_clock.cc:648] HybridClock initialized: now 1790297319852072 us; error 30 us; skew 500 ppm
I20260925 00:48:39.853924 5597 webserver.cc:460] Webserver started at http://127.0.249.126:40787/ using document root <none> and password file <none>
I20260925 00:48:39.854476 5597 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:39.854562 5597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:39.854786 5597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:39.856714 5597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "eb88272cf05b43cd99d43941a9eb570a"
format_stamp: "Formatted at 2026-09-25 00:48:39 on dist-test-slave-05m0"
server_key: "e951327992e447c9e3e8d46bd5dd0843"
server_key_iv: "1d92d6d40f18a229ca6e61078ccac11e"
server_key_version: "encryptionkey@0"
I20260925 00:48:39.857254 5597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "eb88272cf05b43cd99d43941a9eb570a"
format_stamp: "Formatted at 2026-09-25 00:48:39 on dist-test-slave-05m0"
server_key: "e951327992e447c9e3e8d46bd5dd0843"
server_key_iv: "1d92d6d40f18a229ca6e61078ccac11e"
server_key_version: "encryptionkey@0"
I20260925 00:48:39.860661 5597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:48:39.863003 5611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:39.864185 5597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:39.864318 5597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "eb88272cf05b43cd99d43941a9eb570a"
format_stamp: "Formatted at 2026-09-25 00:48:39 on dist-test-slave-05m0"
server_key: "e951327992e447c9e3e8d46bd5dd0843"
server_key_iv: "1d92d6d40f18a229ca6e61078ccac11e"
server_key_version: "encryptionkey@0"
I20260925 00:48:39.864424 5597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:39.868842 5597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:39.869460 5597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:39.869639 5597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:39.876183 5597 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:33771
I20260925 00:48:39.876187 5663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:33771 every 8 connection(s)
I20260925 00:48:39.877039 5597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:39.877933 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 5597
I20260925 00:48:39.878044 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:39.878347 997 external_mini_cluster.cc:1428] Setting key c37b1853b8ce6de3c9c2fe41fff72269
I20260925 00:48:39.880438 5664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:39.884272 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:39.879675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.0.249.126:33771)
0925 00:48:39.880890 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:39.880904 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:39.880937 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:48:39.881629 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:39.881646 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:39.881970 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:39.882300 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:39.883099 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:39.883104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:39.883115 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:39.883125 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:39.883177 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:39.883193 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:39.883339 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:39.883409 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 00:48:39.883638 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":686,"threads_started":1}
I20260925 00:48:39.886211 5664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a: Bootstrap starting.
I20260925 00:48:39.888417 5664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:39.889057 5664 log.cc:824] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:39.890831 5664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a: No bootstrap required, opened a new log
I20260925 00:48:39.894375 5664 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb88272cf05b43cd99d43941a9eb570a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33771 } }
I20260925 00:48:39.894586 5664 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:39.894621 5664 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb88272cf05b43cd99d43941a9eb570a, State: Initialized, Role: FOLLOWER
I20260925 00:48:39.894961 5664 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb88272cf05b43cd99d43941a9eb570a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33771 } }
I20260925 00:48:39.895064 5664 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:39.895107 5664 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:39.895174 5664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:39.895942 5664 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb88272cf05b43cd99d43941a9eb570a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33771 } }
I20260925 00:48:39.896212 5664 leader_election.cc:302] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb88272cf05b43cd99d43941a9eb570a; no voters:
I20260925 00:48:39.896445 5664 leader_election.cc:288] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:39.896586 5669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:39.896826 5669 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [term 1 LEADER]: Becoming Leader. State: Replica: eb88272cf05b43cd99d43941a9eb570a, State: Running, Role: LEADER
I20260925 00:48:39.897135 5669 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb88272cf05b43cd99d43941a9eb570a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33771 } }
I20260925 00:48:39.897675 5664 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:39.898957 5671 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb88272cf05b43cd99d43941a9eb570a. Latest consensus state: current_term: 1 leader_uuid: "eb88272cf05b43cd99d43941a9eb570a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb88272cf05b43cd99d43941a9eb570a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33771 } } }
I20260925 00:48:39.898957 5670 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb88272cf05b43cd99d43941a9eb570a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb88272cf05b43cd99d43941a9eb570a" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33771 } } }
I20260925 00:48:39.899149 5670 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:39.899438 5671 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:39.900223 5679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:39.902705 5679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:39.907380 5679 catalog_manager.cc:1347] Generated new cluster ID: f3cb6bfe473d42c289ebcb6a0e03db3f
I20260925 00:48:39.907464 5679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:39.939195 5679 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:39.939554 5679 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:39.944371 5679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a: Generated new TSK 0
I20260925 00:48:39.944999 5679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:48:39.947490 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33771
--builtin_ntp_servers=127.0.249.84:34123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 00:48:40.060245 5688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:40.060508 5688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:40.060549 5688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:40.064873 5688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:40.064953 5688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:40.065063 5688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:40.070436 5688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:40.071483 5688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:40.072829 5688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:40.082621 5693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:40.082607 5694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:40.082633 5696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:40.082952 5688 server_base.cc:1034] running on GCE node
I20260925 00:48:40.083413 5688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:40.084010 5688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:40.085184 5688 hybrid_clock.cc:648] HybridClock initialized: now 1790297320085149 us; error 47 us; skew 500 ppm
I20260925 00:48:40.087077 5688 webserver.cc:460] Webserver started at http://127.0.249.65:40259/ using document root <none> and password file <none>
I20260925 00:48:40.087651 5688 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:40.087711 5688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:40.087934 5688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:40.090008 5688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "6af39332737342648ac8a411eada3ef6"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "3a9f7ef83cffd0c93ba2192c21ea3471"
server_key_iv: "0737ba039191a107007f38e7e258c0ff"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.090552 5688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "6af39332737342648ac8a411eada3ef6"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "3a9f7ef83cffd0c93ba2192c21ea3471"
server_key_iv: "0737ba039191a107007f38e7e258c0ff"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.094206 5688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:48:40.096494 5702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.097752 5688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:40.097877 5688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "6af39332737342648ac8a411eada3ef6"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "3a9f7ef83cffd0c93ba2192c21ea3471"
server_key_iv: "0737ba039191a107007f38e7e258c0ff"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.098035 5688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:40.102675 5688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:40.103240 5688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:40.103415 5688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:40.103950 5688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:40.104809 5688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:40.104856 5688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.104924 5688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:40.104956 5688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.113327 5688 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:39473
I20260925 00:48:40.113437 5815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:39473 every 8 connection(s)
I20260925 00:48:40.114252 5688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:40.120143 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.115996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:55515 (local address 127.0.249.126:33771)
0925 00:48:40.116159 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.116163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.117143 (+ 980us) server_negotiation.cc:407] Connection header received
0925 00:48:40.118439 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.118442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.118486 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.118567 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.119656 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.119660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.119663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.119675 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.119696 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.119710 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.119800 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.119950 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.119992 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 00:48:40.120913 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.116343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:33771 (local address 127.0.249.65:55515)
0925 00:48:40.116977 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:40.117016 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:40.118228 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:40.118717 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:40.118724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:40.118808 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:40.119394 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:48:40.119395 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:40.119432 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:48:40.119496 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:40.119796 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:40.119805 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:40.119812 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:48:40.119987 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 00:48:40.120239 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":102,"threads_started":1}
I20260925 00:48:40.122172 5816 heartbeater.cc:347] Connected to a master server at 127.0.249.126:33771
I20260925 00:48:40.122395 5816 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:40.122781 5816 heartbeater.cc:511] Master 127.0.249.126:33771 requested a full tablet report, sending...
I20260925 00:48:40.123996 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 5688
I20260925 00:48:40.124137 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:40.124179 5628 ts_manager.cc:194] Registered new tserver with Master: 6af39332737342648ac8a411eada3ef6 (127.0.249.65:39473)
I20260925 00:48:40.124457 997 external_mini_cluster.cc:1428] Setting key 10b554d216d5fae3118833060bc01e5b
I20260925 00:48:40.126609 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33771
--builtin_ntp_servers=127.0.249.84:34123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 00:48:40.237980 5819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:40.238267 5819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:40.238312 5819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:40.242347 5819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:40.242411 5819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:40.242471 5819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:40.247249 5819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:40.248265 5819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:40.249673 5819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:40.259207 5825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:40.259248 5827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:40.259268 5824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:40.259361 5819 server_base.cc:1034] running on GCE node
I20260925 00:48:40.259836 5819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:40.260505 5819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:40.261701 5819 hybrid_clock.cc:648] HybridClock initialized: now 1790297320261679 us; error 26 us; skew 500 ppm
I20260925 00:48:40.263479 5819 webserver.cc:460] Webserver started at http://127.0.249.66:36667/ using document root <none> and password file <none>
I20260925 00:48:40.264042 5819 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:40.264142 5819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:40.264364 5819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:40.266573 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "6011788666f64bb6a23bd0048022f378"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "21c008611c2efcd7cf1e37970c595249"
server_key_iv: "2ef8ca96442e7a97520598c2cef0182c"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.267130 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "6011788666f64bb6a23bd0048022f378"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "21c008611c2efcd7cf1e37970c595249"
server_key_iv: "2ef8ca96442e7a97520598c2cef0182c"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.270777 5819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:48:40.273006 5833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.274211 5819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:40.274341 5819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "6011788666f64bb6a23bd0048022f378"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "21c008611c2efcd7cf1e37970c595249"
server_key_iv: "2ef8ca96442e7a97520598c2cef0182c"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.274461 5819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:40.278880 5819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:40.279477 5819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:40.279680 5819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:40.280208 5819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:40.281064 5819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:40.281133 5819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.281198 5819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:40.281245 5819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.290033 5819 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:44683
I20260925 00:48:40.290055 5946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:44683 every 8 connection(s)
I20260925 00:48:40.290894 5819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:40.292440 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 5819
I20260925 00:48:40.292579 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:40.292908 997 external_mini_cluster.cc:1428] Setting key 0bea224b3604d6fde5341dbd26737863
I20260925 00:48:40.295122 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33771
--builtin_ntp_servers=127.0.249.84:34123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 00:48:40.297614 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.292613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:52247 (local address 127.0.249.126:33771)
0925 00:48:40.292771 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.292777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.293950 (+ 1173us) server_negotiation.cc:407] Connection header received
0925 00:48:40.295673 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.295676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.295710 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.295784 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.297057 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.297063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.297068 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.297084 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.297112 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.297124 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.297224 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.297348 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.297449 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:48:40.298413 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.293040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:33771 (local address 127.0.249.66:52247)
0925 00:48:40.293817 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:40.293856 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:40.295434 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:40.295952 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:40.295963 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:40.296089 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:40.296677 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:48:40.296679 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:40.296731 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:48:40.296807 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:40.297215 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:40.297221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:40.297225 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 00:48:40.297356 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 00:48:40.297649 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":141,"threads_started":1}
I20260925 00:48:40.301002 5947 heartbeater.cc:347] Connected to a master server at 127.0.249.126:33771
I20260925 00:48:40.301256 5947 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:40.301788 5947 heartbeater.cc:511] Master 127.0.249.126:33771 requested a full tablet report, sending...
I20260925 00:48:40.302948 5627 ts_manager.cc:194] Registered new tserver with Master: 6011788666f64bb6a23bd0048022f378 (127.0.249.66:44683)
W20260925 00:48:40.406967 5950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:40.407223 5950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:40.407263 5950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:40.411268 5950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:40.411326 5950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:40.411381 5950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:40.416128 5950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:40.417160 5950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:40.418574 5950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:40.428297 5958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:40.428288 5956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:40.428357 5950 server_base.cc:1034] running on GCE node
W20260925 00:48:40.428283 5955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:40.428890 5950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:40.429605 5950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:40.430759 5950 hybrid_clock.cc:648] HybridClock initialized: now 1790297320430728 us; error 44 us; skew 500 ppm
I20260925 00:48:40.432607 5950 webserver.cc:460] Webserver started at http://127.0.249.67:36433/ using document root <none> and password file <none>
I20260925 00:48:40.433168 5950 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:40.433229 5950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:40.433506 5950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:40.435508 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "f241e89648264a828b0fc036ca2be4d9"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "6610fde289e41b911345d23675a7b9cf"
server_key_iv: "0293591b8ca4c5eab5cfb0251978f8ad"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.436065 5950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "f241e89648264a828b0fc036ca2be4d9"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "6610fde289e41b911345d23675a7b9cf"
server_key_iv: "0293591b8ca4c5eab5cfb0251978f8ad"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.439953 5950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:48:40.442482 5964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.443754 5950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 00:48:40.443882 5950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "f241e89648264a828b0fc036ca2be4d9"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "6610fde289e41b911345d23675a7b9cf"
server_key_iv: "0293591b8ca4c5eab5cfb0251978f8ad"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.444005 5950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:40.448980 5950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:40.449679 5950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:40.449882 5950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:40.450460 5950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:40.451403 5950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:40.451455 5950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.451498 5950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:40.451555 5950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.461010 5950 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:41361
I20260925 00:48:40.461030 6077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:41361 every 8 connection(s)
I20260925 00:48:40.462028 5950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:40.462455 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 5950
I20260925 00:48:40.462572 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:40.462857 997 external_mini_cluster.cc:1428] Setting key 4c3ad7c8a3ce31bb396ff81c5f8d93e5
I20260925 00:48:40.468781 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.464109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:42217 (local address 127.0.249.126:33771)
0925 00:48:40.464274 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.464279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.465665 (+ 1386us) server_negotiation.cc:407] Connection header received
0925 00:48:40.467061 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.467064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.467104 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.467200 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.468294 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.468297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.468300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.468312 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.468333 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.468354 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.468447 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.468594 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.468638 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:48:40.469714 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.464682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:33771 (local address 127.0.249.67:42217)
0925 00:48:40.465532 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:40.465572 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:40.466836 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:40.467384 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:40.467391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:40.467479 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:40.468028 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:48:40.468030 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:40.468070 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:48:40.468135 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:40.468435 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:40.468443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:40.468448 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:48:40.468633 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 00:48:40.468966 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":122,"threads_started":1}
I20260925 00:48:40.470741 6078 heartbeater.cc:347] Connected to a master server at 127.0.249.126:33771
I20260925 00:48:40.470988 6078 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:40.471398 6078 heartbeater.cc:511] Master 127.0.249.126:33771 requested a full tablet report, sending...
I20260925 00:48:40.472425 5627 ts_manager.cc:194] Registered new tserver with Master: f241e89648264a828b0fc036ca2be4d9 (127.0.249.67:41361)
I20260925 00:48:40.476924 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:48:40.480504 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.478943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.0.249.126:33771)
0925 00:48:40.479100 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.479109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.479370 (+ 261us) server_negotiation.cc:407] Connection header received
0925 00:48:40.479457 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.479459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.479488 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.479559 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.480246 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.480249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.480251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.480259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.480274 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.480281 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.480347 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.480387 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.480415 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 00:48:40.483206 5627 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:40.486522 5627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:40.497645 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.494616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:39473 (local address 127.0.0.1:33256)
0925 00:48:40.494991 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:40.495058 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:40.495151 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:40.496137 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:40.496144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:40.496219 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:40.496797 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:48:40.496801 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:40.496824 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:48:40.496891 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:40.497264 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:40.497274 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:40.497280 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:48:40.497420 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 00:48:40.497500 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20260925 00:48:40.498351 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.495418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:41361 (local address 127.0.0.1:46958)
0925 00:48:40.496298 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:40.496313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:40.496388 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:40.497086 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:40.497089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:40.497107 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:40.497539 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:48:40.497541 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:40.497551 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:48:40.497576 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:40.498067 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:40.498073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:40.498080 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:48:40.498166 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 00:48:40.498207 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":81,"threads_started":1}
I20260925 00:48:40.498600 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.494941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:44683 (local address 127.0.0.1:51604)
0925 00:48:40.495797 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:40.495810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:40.495890 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:40.497539 (+ 1649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:40.497542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:40.497557 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:40.497964 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:48:40.497967 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:40.497979 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:48:40.498008 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:40.498371 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:40.498373 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:40.498376 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 00:48:40.498448 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:48:40.498484 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":105,"threads_started":1}
I20260925 00:48:40.498687 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.494810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.0.249.65:39473)
0925 00:48:40.495559 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.495572 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.495600 (+ 28us) server_negotiation.cc:407] Connection header received
0925 00:48:40.495662 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.495667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.495967 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.496131 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.497029 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.497040 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.497055 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.497067 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.497115 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.497147 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.498364 (+ 1217us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.498422 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.498507 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":130,"threads_started":1}
I20260925 00:48:40.498818 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.495829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.0.249.67:41361)
0925 00:48:40.496676 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.496688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.496713 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:48:40.496765 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.496770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.496949 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.497095 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.497740 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.497747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.497759 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.497772 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.497813 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.497841 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.497967 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.498578 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.498644 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":150,"threads_started":1}
I20260925 00:48:40.499114 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.495442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.0.249.66:44683)
0925 00:48:40.496116 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.496128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.496150 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:48:40.496195 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.496200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.496365 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.496858 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.498141 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.498148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.498162 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.498173 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.498210 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.498236 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.498332 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.498864 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.498937 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":93,"threads_started":1}
I20260925 00:48:40.501291 5750 tablet_service.cc:1467] Processing CreateTablet for tablet 816a907305384637a1f1a6c18315ed60 (DEFAULT_TABLE table=test-table [id=5c6ba3be446641c5af4ecb0c6769fc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:40.501748 5881 tablet_service.cc:1467] Processing CreateTablet for tablet 816a907305384637a1f1a6c18315ed60 (DEFAULT_TABLE table=test-table [id=5c6ba3be446641c5af4ecb0c6769fc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:40.501751 6012 tablet_service.cc:1467] Processing CreateTablet for tablet 816a907305384637a1f1a6c18315ed60 (DEFAULT_TABLE table=test-table [id=5c6ba3be446641c5af4ecb0c6769fc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:40.502498 5750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816a907305384637a1f1a6c18315ed60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:40.502761 6012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816a907305384637a1f1a6c18315ed60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:40.502761 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816a907305384637a1f1a6c18315ed60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:40.508422 6095 tablet_bootstrap.cc:492] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6: Bootstrap starting.
I20260925 00:48:40.510011 6096 tablet_bootstrap.cc:492] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: Bootstrap starting.
I20260925 00:48:40.510980 6097 tablet_bootstrap.cc:492] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: Bootstrap starting.
I20260925 00:48:40.511438 6095 tablet_bootstrap.cc:654] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:40.512109 6096 tablet_bootstrap.cc:654] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:40.512302 6095 log.cc:824] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:40.512888 6096 log.cc:824] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:40.512975 6097 tablet_bootstrap.cc:654] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:40.513849 6097 log.cc:824] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:40.514508 6095 tablet_bootstrap.cc:492] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6: No bootstrap required, opened a new log
I20260925 00:48:40.514707 6095 ts_tablet_manager.cc:1397] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260925 00:48:40.514827 6096 tablet_bootstrap.cc:492] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: No bootstrap required, opened a new log
I20260925 00:48:40.514990 6096 ts_tablet_manager.cc:1397] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:48:40.515702 6097 tablet_bootstrap.cc:492] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: No bootstrap required, opened a new log
I20260925 00:48:40.515903 6097 ts_tablet_manager.cc:1397] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:48:40.519567 6095 raft_consensus.cc:348] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } }
I20260925 00:48:40.519567 6096 raft_consensus.cc:348] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } }
I20260925 00:48:40.519830 6096 raft_consensus.cc:374] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:40.519830 6095 raft_consensus.cc:374] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:40.519690 6097 raft_consensus.cc:348] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } }
I20260925 00:48:40.519888 6095 raft_consensus.cc:729] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af39332737342648ac8a411eada3ef6, State: Initialized, Role: FOLLOWER
I20260925 00:48:40.519888 6096 raft_consensus.cc:729] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f241e89648264a828b0fc036ca2be4d9, State: Initialized, Role: FOLLOWER
I20260925 00:48:40.519944 6097 raft_consensus.cc:374] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:40.520008 6097 raft_consensus.cc:729] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6011788666f64bb6a23bd0048022f378, State: Initialized, Role: FOLLOWER
I20260925 00:48:40.520365 6096 consensus_queue.cc:260] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } }
I20260925 00:48:40.520365 6095 consensus_queue.cc:260] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } }
I20260925 00:48:40.520455 6097 consensus_queue.cc:260] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } }
I20260925 00:48:40.521016 5816 heartbeater.cc:503] Master 127.0.249.126:33771 was elected leader, sending a full tablet report...
I20260925 00:48:40.521339 6095 ts_tablet_manager.cc:1428] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 00:48:40.521502 6097 ts_tablet_manager.cc:1428] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260925 00:48:40.521754 5947 heartbeater.cc:503] Master 127.0.249.126:33771 was elected leader, sending a full tablet report...
I20260925 00:48:40.521899 6078 heartbeater.cc:503] Master 127.0.249.126:33771 was elected leader, sending a full tablet report...
I20260925 00:48:40.522421 6096 ts_tablet_manager.cc:1428] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260925 00:48:40.541814 5948 tablet.cc:2369] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:48:40.615365 5817 tablet.cc:2369] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:40.682036 6101 raft_consensus.cc:482] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:40.682214 6101 raft_consensus.cc:504] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } }
I20260925 00:48:40.683254 6101 leader_election.cc:288] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af39332737342648ac8a411eada3ef6 (127.0.249.65:39473), 6011788666f64bb6a23bd0048022f378 (127.0.249.66:44683)
I20260925 00:48:40.685111 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.683441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:39473 (local address 127.0.249.67:50575)
0925 00:48:40.683641 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:40.683656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:40.683736 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:40.684002 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:40.684004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:40.684018 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:40.684553 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:48:40.684555 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:40.684564 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:48:40.684598 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:40.684849 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:40.684852 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:40.684856 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 00:48:40.684954 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 00:48:40.684991 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260925 00:48:40.685115 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.683521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:50575 (local address 127.0.249.65:39473)
0925 00:48:40.683642 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.683648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.683774 (+ 126us) server_negotiation.cc:407] Connection header received
0925 00:48:40.683829 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.683832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.683877 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.683972 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.684719 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.684727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.684728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.684739 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.684756 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.684764 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.684853 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.684958 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.684995 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:48:40.685339 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.683647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:51875 (local address 127.0.249.66:44683)
0925 00:48:40.683768 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.683775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.683926 (+ 151us) server_negotiation.cc:407] Connection header received
0925 00:48:40.684016 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.684018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.684072 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.684163 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.684943 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.684951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.684953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.684963 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.684979 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.684986 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.685083 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.685160 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.685206 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 00:48:40.685621 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.683489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:44683 (local address 127.0.249.67:51875)
0925 00:48:40.683832 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:40.683845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:40.683935 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:40.684225 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:40.684230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:40.684249 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:40.684749 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:48:40.684751 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:40.684760 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:48:40.684783 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:40.685068 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:40.685071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:40.685074 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 00:48:40.685149 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 00:48:40.685476 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260925 00:48:40.685791 5770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816a907305384637a1f1a6c18315ed60" candidate_uuid: "f241e89648264a828b0fc036ca2be4d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af39332737342648ac8a411eada3ef6" is_pre_election: true
I20260925 00:48:40.686092 5770 raft_consensus.cc:2454] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f241e89648264a828b0fc036ca2be4d9 in term 0.
I20260925 00:48:40.686206 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816a907305384637a1f1a6c18315ed60" candidate_uuid: "f241e89648264a828b0fc036ca2be4d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6011788666f64bb6a23bd0048022f378" is_pre_election: true
I20260925 00:48:40.686504 5901 raft_consensus.cc:2454] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f241e89648264a828b0fc036ca2be4d9 in term 0.
I20260925 00:48:40.686528 5967 leader_election.cc:302] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af39332737342648ac8a411eada3ef6, f241e89648264a828b0fc036ca2be4d9; no voters:
I20260925 00:48:40.686808 6101 raft_consensus.cc:2781] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:40.686934 6101 raft_consensus.cc:482] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:40.686995 6101 raft_consensus.cc:3037] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:40.687978 6101 raft_consensus.cc:504] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } }
I20260925 00:48:40.688300 6101 leader_election.cc:288] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [CANDIDATE]: Term 1 election: Requested vote from peers 6af39332737342648ac8a411eada3ef6 (127.0.249.65:39473), 6011788666f64bb6a23bd0048022f378 (127.0.249.66:44683)
I20260925 00:48:40.688652 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816a907305384637a1f1a6c18315ed60" candidate_uuid: "f241e89648264a828b0fc036ca2be4d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6011788666f64bb6a23bd0048022f378"
I20260925 00:48:40.688643 5770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816a907305384637a1f1a6c18315ed60" candidate_uuid: "f241e89648264a828b0fc036ca2be4d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af39332737342648ac8a411eada3ef6"
I20260925 00:48:40.688766 5770 raft_consensus.cc:3037] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:40.688766 5901 raft_consensus.cc:3037] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:40.690022 5901 raft_consensus.cc:2454] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f241e89648264a828b0fc036ca2be4d9 in term 1.
I20260925 00:48:40.690022 5770 raft_consensus.cc:2454] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f241e89648264a828b0fc036ca2be4d9 in term 1.
I20260925 00:48:40.690276 5967 leader_election.cc:302] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6011788666f64bb6a23bd0048022f378, 6af39332737342648ac8a411eada3ef6, f241e89648264a828b0fc036ca2be4d9; no voters:
I20260925 00:48:40.690465 6101 raft_consensus.cc:2781] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:40.690631 6101 raft_consensus.cc:686] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 1 LEADER]: Becoming Leader. State: Replica: f241e89648264a828b0fc036ca2be4d9, State: Running, Role: LEADER
I20260925 00:48:40.690940 6101 consensus_queue.cc:237] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } }
I20260925 00:48:40.694074 5626 catalog_manager.cc:5495] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f241e89648264a828b0fc036ca2be4d9 (127.0.249.67). New cstate: current_term: 1 leader_uuid: "f241e89648264a828b0fc036ca2be4d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f241e89648264a828b0fc036ca2be4d9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 41361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 } health_report { overall_health: UNKNOWN } } }
W20260925 00:48:40.713056 6079 tablet.cc:2369] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:40.732743 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.731477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.0.249.67:41361)
0925 00:48:40.731607 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.731611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.731638 (+ 27us) server_negotiation.cc:407] Connection header received
0925 00:48:40.731708 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.731710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.731764 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.731826 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.732440 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.732442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.732445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.732455 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.732473 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.732480 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.732552 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.732615 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.732646 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:48:40.738446 5901 raft_consensus.cc:1261] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [term 1 FOLLOWER]: Refusing update from remote peer f241e89648264a828b0fc036ca2be4d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:40.738452 5770 raft_consensus.cc:1261] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6 [term 1 FOLLOWER]: Refusing update from remote peer f241e89648264a828b0fc036ca2be4d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:40.739176 6105 consensus_queue.cc:1035] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6011788666f64bb6a23bd0048022f378" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:40.739372 6101 consensus_queue.cc:1035] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af39332737342648ac8a411eada3ef6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:40.747320 6109 mvcc.cc:205] Tried to move back new op lower bound from 7333057825737482240 to 7333057825552551936. Current Snapshot: MvccSnapshot[applied={T|T < 7333057825737482240}]
I20260925 00:48:40.756260 5626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:40.756453 5626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:40.758865 5626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb88272cf05b43cd99d43941a9eb570a: Sending DeleteTablet for 3 replicas of tablet 816a907305384637a1f1a6c18315ed60
I20260925 00:48:40.759676 6012 tablet_service.cc:1514] Processing DeleteTablet for tablet 816a907305384637a1f1a6c18315ed60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:40 UTC) from {username='slave'} at 127.0.0.1:46958
I20260925 00:48:40.759676 5750 tablet_service.cc:1514] Processing DeleteTablet for tablet 816a907305384637a1f1a6c18315ed60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:40 UTC) from {username='slave'} at 127.0.0.1:33256
I20260925 00:48:40.759832 5881 tablet_service.cc:1514] Processing DeleteTablet for tablet 816a907305384637a1f1a6c18315ed60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:40 UTC) from {username='slave'} at 127.0.0.1:51604
I20260925 00:48:40.760247 6119 tablet_replica.cc:330] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6: stopping tablet replica
I20260925 00:48:40.760375 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 5688
I20260925 00:48:40.760579 6119 raft_consensus.cc:2229] T 816a907305384637a1f1a6c18315ed60 P 6af39332737342648ac8a411eada3ef6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:40.760987 6120 tablet_replica.cc:330] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: stopping tablet replica
I20260925 00:48:40.761250 6120 raft_consensus.cc:2229] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:40.761555 6120 raft_consensus.cc:2258] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:40.763275 6120 ts_tablet_manager.cc:1905] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:40.765446 6120 ts_tablet_manager.cc:1918] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:40.765548 6120 log.cc:1196] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/816a907305384637a1f1a6c18315ed60
I20260925 00:48:40.765750 6118 tablet_replica.cc:330] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: stopping tablet replica
I20260925 00:48:40.765863 6120 ts_tablet_manager.cc:1939] T 816a907305384637a1f1a6c18315ed60 P 6011788666f64bb6a23bd0048022f378: Deleting consensus metadata
I20260925 00:48:40.766136 6118 raft_consensus.cc:2229] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:40.766561 6118 raft_consensus.cc:2258] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:40.767139 5613 catalog_manager.cc:4841] TS 6011788666f64bb6a23bd0048022f378 (127.0.249.66:44683): tablet 816a907305384637a1f1a6c18315ed60 (table test-table [id=5c6ba3be446641c5af4ecb0c6769fc24]) successfully deleted
I20260925 00:48:40.768101 6118 ts_tablet_manager.cc:1905] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:48:40.768971 5614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:39473 (error 108)
I20260925 00:48:40.770042 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.769713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:39473 (local address 127.0.0.1:33262)
0925 00:48:40.769862 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:40.769945 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 00:48:40.770038 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 5819
I20260925 00:48:40.770952 6118 ts_tablet_manager.cc:1918] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:40.771059 6118 log.cc:1196] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/816a907305384637a1f1a6c18315ed60
I20260925 00:48:40.771377 6118 ts_tablet_manager.cc:1939] T 816a907305384637a1f1a6c18315ed60 P f241e89648264a828b0fc036ca2be4d9: Deleting consensus metadata
W20260925 00:48:40.772799 5614 catalog_manager.cc:4584] TS 6af39332737342648ac8a411eada3ef6 (127.0.249.65:39473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 816a907305384637a1f1a6c18315ed60: Network error: Client connection negotiation failed: client connection to 127.0.249.65:39473: connect: Connection refused (error 111)
I20260925 00:48:40.773003 5614 catalog_manager.cc:4841] TS f241e89648264a828b0fc036ca2be4d9 (127.0.249.67:41361): tablet 816a907305384637a1f1a6c18315ed60 (table test-table [id=5c6ba3be446641c5af4ecb0c6769fc24]) successfully deleted
I20260925 00:48:40.778378 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 5950
I20260925 00:48:40.783711 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 5597
2026-09-25T00:48:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1089 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 00:48:40.804785 997 mini_oidc.cc:152] Starting JWKS server
I20260925 00:48:40.806438 997 webserver.cc:460] Webserver started at https://127.0.0.1:45367/ using document root <none> and password file <none>
I20260925 00:48:40.806623 997 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 00:48:40.806886 997 webserver.cc:460] Webserver started at http://127.0.0.1:35107/ using document root <none> and password file <none>
2026-09-25T00:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:40Z Disabled control of system clock
I20260925 00:48:40.815578 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46405
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46405
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45367/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/testchainca.cert with env {}
W20260925 00:48:40.924167 6127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:40.924417 6127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:40.924456 6127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:40.928423 6127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:40.928500 6127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:40.928515 6127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:40.928531 6127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:40.928546 6127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 00:48:40.928561 6127 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 00:48:40.928575 6127 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45367/jwks.json
I20260925 00:48:40.933966 6127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45367/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:40.934923 6127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:40.935840 6127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:40.944062 6135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:40.944164 6132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:40.944064 6133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:40.944314 6127 server_base.cc:1034] running on GCE node
I20260925 00:48:40.944968 6127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:40.946141 6127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:40.947324 6127 hybrid_clock.cc:648] HybridClock initialized: now 1790297320947297 us; error 36 us; skew 500 ppm
I20260925 00:48:40.949162 6127 webserver.cc:460] Webserver started at http://127.0.249.126:40731/ using document root <none> and password file <none>
I20260925 00:48:40.949782 6127 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:40.949846 6127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:40.950078 6127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:40.952035 6127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "1a771df4489341daa047683af08b4135"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "a1b443367a74996277ff4f07d84e73fb"
server_key_iv: "ceb262141a0a7c070170e5e204c71ab7"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.952582 6127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "1a771df4489341daa047683af08b4135"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "a1b443367a74996277ff4f07d84e73fb"
server_key_iv: "ceb262141a0a7c070170e5e204c71ab7"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.956040 6127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 00:48:40.958338 6141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:40.959517 6127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:40.959653 6127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "1a771df4489341daa047683af08b4135"
format_stamp: "Formatted at 2026-09-25 00:48:40 on dist-test-slave-05m0"
server_key: "a1b443367a74996277ff4f07d84e73fb"
server_key_iv: "ceb262141a0a7c070170e5e204c71ab7"
server_key_version: "encryptionkey@0"
I20260925 00:48:40.959762 6127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:40.972736 6127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:40.973378 6127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:40.973654 6127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:40.980209 6193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:46405 every 8 connection(s)
I20260925 00:48:40.980207 6127 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:46405
I20260925 00:48:40.981208 6127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:40.984306 6194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:40.989619 6194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135: Bootstrap starting.
I20260925 00:48:40.991106 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 6127
I20260925 00:48:40.991246 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:40.991511 997 external_mini_cluster.cc:1428] Setting key 8b9e691c505eb3485dd5652df26459d1
I20260925 00:48:40.992138 6194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:40.992998 6194 log.cc:824] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:40.994887 6194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135: No bootstrap required, opened a new log
I20260925 00:48:40.998390 6194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a771df4489341daa047683af08b4135" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46405 } }
I20260925 00:48:40.998641 6194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:40.998725 6194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a771df4489341daa047683af08b4135, State: Initialized, Role: FOLLOWER
I20260925 00:48:40.999207 6194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a771df4489341daa047683af08b4135" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46405 } }
I20260925 00:48:40.999349 6194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:40.999439 6194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:40.999558 6194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:41.000451 6194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a771df4489341daa047683af08b4135" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46405 } }
I20260925 00:48:41.000626 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:40.992575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.0.249.126:46405)
0925 00:48:40.993237 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:40.993252 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:40.993287 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:48:40.994105 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:40.994125 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:40.994449 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:40.994788 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:40.995199 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:40.996142 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:40.997051 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:40.997578 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:40.997746 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:40.998609 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:40.998614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:40.998627 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:40.999110 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:40.999164 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:40.999184 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:40.999293 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:40.999501 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 00:48:40.999895 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260925 00:48:41.000862 6194 leader_election.cc:302] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a771df4489341daa047683af08b4135; no voters:
I20260925 00:48:41.001183 6194 leader_election.cc:288] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:41.001291 6199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:41.001571 6199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [term 1 LEADER]: Becoming Leader. State: Replica: 1a771df4489341daa047683af08b4135, State: Running, Role: LEADER
I20260925 00:48:41.001869 6199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a771df4489341daa047683af08b4135" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46405 } }
I20260925 00:48:41.003207 6194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:41.003561 6200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a771df4489341daa047683af08b4135. Latest consensus state: current_term: 1 leader_uuid: "1a771df4489341daa047683af08b4135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a771df4489341daa047683af08b4135" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46405 } } }
I20260925 00:48:41.003672 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:48:41.003683 6200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:41.003963 6201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a771df4489341daa047683af08b4135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a771df4489341daa047683af08b4135" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46405 } } }
I20260925 00:48:41.004515 6201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:41.004892 6203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:41.007680 6203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:41.013025 6203 catalog_manager.cc:1347] Generated new cluster ID: f8f00a79704549e28f33f01f0b0ecb15
I20260925 00:48:41.013087 6203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:41.027163 6203 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:41.028367 6203 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:41.033888 6203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a771df4489341daa047683af08b4135: Generated new TSK 0
I20260925 00:48:41.034670 6203 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 00:48:41.064235 6224 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 00:48:41.064405 6224 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 00:48:41.065740 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:41.047863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.249.126:46405)
0925 00:48:41.047997 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:41.048002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:41.048233 (+ 231us) server_negotiation.cc:407] Connection header received
0925 00:48:41.048416 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:41.048419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:41.048474 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:41.048584 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0925 00:48:41.048811 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.049667 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.050511 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.050858 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.050957 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.051263 (+ 306us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 00:48:41.064374 (+ 13111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 00:48:41.065638 (+ 1264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 00:48:41.065836 6197 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 00:48:41.066124 6223 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:41.047772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46405 (local address 127.0.0.1:45894)
0925 00:48:41.048154 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:41.048166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:41.048269 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:41.048600 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:41.048602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:41.048608 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 00:48:41.048715 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:41.048719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.049790 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.049793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:41.050405 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:41.050412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.050966 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.050969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:41.051036 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.051085 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 00:48:41.064622 (+ 13537us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 00:48:41.064827 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.249.126:46405: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.249.126:46405: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 00:48:41.066982 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 6127
2026-09-25T00:48:41Z chronyd exiting
I20260925 00:48:41.104177 997 test_util.cc:175] -----------------------------------------------
I20260925 00:48:41.104259 997 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790297292685301-997-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 00:48:41.105288 997 mini_oidc.cc:152] Starting JWKS server
I20260925 00:48:41.106169 997 webserver.cc:460] Webserver started at https://127.0.0.1:40369/ using document root <none> and password file <none>
I20260925 00:48:41.106283 997 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 00:48:41.106549 997 webserver.cc:460] Webserver started at http://127.0.0.1:34841/ using document root <none> and password file <none>
2026-09-25T00:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:41Z Disabled control of system clock
I20260925 00:48:41.115181 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:45879
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:41191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:45879
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40369/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/testchainca.cert with env {}
W20260925 00:48:41.224509 6230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:41.224805 6230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:41.224846 6230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:41.229206 6230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:41.229279 6230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:41.229295 6230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:41.229312 6230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:41.229328 6230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 00:48:41.229388 6230 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 00:48:41.229457 6230 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40369/jwks.json
I20260925 00:48:41.234982 6230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:41191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:45879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:45879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40369/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:41.236024 6230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:41.237025 6230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:41.245702 6235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:41.245774 6236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:41.245709 6238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:41.245997 6230 server_base.cc:1034] running on GCE node
I20260925 00:48:41.246605 6230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:41.247699 6230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:41.248880 6230 hybrid_clock.cc:648] HybridClock initialized: now 1790297321248850 us; error 46 us; skew 500 ppm
I20260925 00:48:41.250881 6230 webserver.cc:460] Webserver started at http://127.0.249.126:37501/ using document root <none> and password file <none>
I20260925 00:48:41.251452 6230 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:41.251541 6230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:41.251771 6230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:41.253815 6230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689"
format_stamp: "Formatted at 2026-09-25 00:48:41 on dist-test-slave-05m0"
server_key: "800376194d28debdb231569b663eb852"
server_key_iv: "7d21c3356ec806971d2051a93fe8f152"
server_key_version: "encryptionkey@0"
I20260925 00:48:41.254369 6230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689"
format_stamp: "Formatted at 2026-09-25 00:48:41 on dist-test-slave-05m0"
server_key: "800376194d28debdb231569b663eb852"
server_key_iv: "7d21c3356ec806971d2051a93fe8f152"
server_key_version: "encryptionkey@0"
I20260925 00:48:41.257994 6230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:41.260329 6244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:41.261641 6230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:41.261773 6230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689"
format_stamp: "Formatted at 2026-09-25 00:48:41 on dist-test-slave-05m0"
server_key: "800376194d28debdb231569b663eb852"
server_key_iv: "7d21c3356ec806971d2051a93fe8f152"
server_key_version: "encryptionkey@0"
I20260925 00:48:41.261883 6230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:41.284180 6230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:41.284907 6230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:41.285108 6230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:41.293038 6230 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:45879
I20260925 00:48:41.293051 6296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:45879 every 8 connection(s)
I20260925 00:48:41.294104 6230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:41.297384 6297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:41.300835 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 6230
I20260925 00:48:41.300988 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:41.301359 997 external_mini_cluster.cc:1428] Setting key aa295c336702f497981b7cb14c149278
I20260925 00:48:41.303859 6297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689: Bootstrap starting.
I20260925 00:48:41.306249 6297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:41.307145 6297 log.cc:824] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:41.309347 6297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689: No bootstrap required, opened a new log
I20260925 00:48:41.311744 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:41.302870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.0.249.126:45879)
0925 00:48:41.303557 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:41.303573 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:41.303611 (+ 38us) server_negotiation.cc:407] Connection header received
0925 00:48:41.304338 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:41.304357 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:41.304764 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:41.305109 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:41.305591 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.306632 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.307671 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.308334 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.308504 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.309473 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:41.309480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:41.309497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:41.310074 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:41.310145 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:41.310173 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:41.310282 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:41.310479 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 00:48:41.310755 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":143,"threads_started":1}
I20260925 00:48:41.314277 6297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45879 } }
I20260925 00:48:41.314575 6297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:41.314663 6297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fa435fdb5b4f8e9d88bd32c4a8a689, State: Initialized, Role: FOLLOWER
I20260925 00:48:41.314934 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:48:41.315169 6297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45879 } }
I20260925 00:48:41.315333 6297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:41.315440 6297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:41.315553 6297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:41.316828 6297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45879 } }
I20260925 00:48:41.317184 6297 leader_election.cc:302] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99fa435fdb5b4f8e9d88bd32c4a8a689; no voters:
I20260925 00:48:41.317530 6297 leader_election.cc:288] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:41.317663 6302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:41.317971 6302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [term 1 LEADER]: Becoming Leader. State: Replica: 99fa435fdb5b4f8e9d88bd32c4a8a689, State: Running, Role: LEADER
I20260925 00:48:41.318320 6302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45879 } }
I20260925 00:48:41.318830 6297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:41.320065 6303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45879 } } }
I20260925 00:48:41.320225 6303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:41.320824 6311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:41.321807 6304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99fa435fdb5b4f8e9d88bd32c4a8a689. Latest consensus state: current_term: 1 leader_uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa435fdb5b4f8e9d88bd32c4a8a689" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45879 } } }
I20260925 00:48:41.322563 6304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:41.324542 6311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:41.330992 6311 catalog_manager.cc:1347] Generated new cluster ID: b6b4493988b54cbe846d5be5574e3c0b
I20260925 00:48:41.331113 6311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:41.342391 6311 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:41.343355 6311 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:41.349062 6311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99fa435fdb5b4f8e9d88bd32c4a8a689: Generated new TSK 0
I20260925 00:48:41.349831 6311 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 00:48:41.382910 6327 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 00:48:41.383008 6327 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 00:48:41.383589 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:41.366525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.0.249.126:45879)
0925 00:48:41.366641 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:41.366646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:41.366840 (+ 194us) server_negotiation.cc:407] Connection header received
0925 00:48:41.366980 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:41.366984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:41.367040 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:41.367161 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0925 00:48:41.367417 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.368251 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.369140 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.369538 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.369640 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.369937 (+ 297us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 00:48:41.383213 (+ 13276us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 00:48:41.383452 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 00:48:41.383651 6300 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 00:48:41.383692 6326 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:41.366466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:45879 (local address 127.0.0.1:47618)
0925 00:48:41.366753 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:41.366765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:41.366857 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:41.367193 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:41.367196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:41.367204 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 00:48:41.367304 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:41.367308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.368374 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.368377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:41.369006 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:41.369013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.369675 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.369678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:41.369744 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.369765 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 00:48:41.383406 (+ 13641us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 00:48:41.383529 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.249.126:45879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":102,"threads_started":1}
I20260925 00:48:41.384617 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 6230
2026-09-25T00:48:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 00:48:41.423549 997 mini_oidc.cc:152] Starting JWKS server
I20260925 00:48:41.424399 997 webserver.cc:460] Webserver started at https://127.0.0.1:36753/ using document root <none> and password file <none>
I20260925 00:48:41.424535 997 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 00:48:41.424811 997 webserver.cc:460] Webserver started at http://127.0.0.1:34377/ using document root <none> and password file <none>
2026-09-25T00:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:41Z Disabled control of system clock
I20260925 00:48:41.433233 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:35821
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:35821
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36753/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 00:48:41.542455 6333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:41.542747 6333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:41.542784 6333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:41.546824 6333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:41.546890 6333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:41.546905 6333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:41.546921 6333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:41.546937 6333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 00:48:41.546998 6333 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 00:48:41.547025 6333 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36753/jwks.json
I20260925 00:48:41.551729 6333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:35821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:35821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36753/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:41.552784 6333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:41.554203 6333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:41.562228 6339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:41.562304 6338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:41.562273 6341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:41.562574 6333 server_base.cc:1034] running on GCE node
I20260925 00:48:41.563126 6333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:41.564226 6333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:41.565454 6333 hybrid_clock.cc:648] HybridClock initialized: now 1790297321565411 us; error 53 us; skew 500 ppm
I20260925 00:48:41.567175 6333 webserver.cc:460] Webserver started at http://127.0.249.126:39823/ using document root <none> and password file <none>
I20260925 00:48:41.567795 6333 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:41.567884 6333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:41.568089 6333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:41.570194 6333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "fddff4588ba54d99be967a29766a2d68"
format_stamp: "Formatted at 2026-09-25 00:48:41 on dist-test-slave-05m0"
server_key: "87bc504857607cb9e42adc99fb34320a"
server_key_iv: "f024ed02e20529cb7ae8ee1292767012"
server_key_version: "encryptionkey@0"
I20260925 00:48:41.570818 6333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "fddff4588ba54d99be967a29766a2d68"
format_stamp: "Formatted at 2026-09-25 00:48:41 on dist-test-slave-05m0"
server_key: "87bc504857607cb9e42adc99fb34320a"
server_key_iv: "f024ed02e20529cb7ae8ee1292767012"
server_key_version: "encryptionkey@0"
I20260925 00:48:41.574357 6333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:48:41.576679 6347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:41.577942 6333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:41.578069 6333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "fddff4588ba54d99be967a29766a2d68"
format_stamp: "Formatted at 2026-09-25 00:48:41 on dist-test-slave-05m0"
server_key: "87bc504857607cb9e42adc99fb34320a"
server_key_iv: "f024ed02e20529cb7ae8ee1292767012"
server_key_version: "encryptionkey@0"
I20260925 00:48:41.578193 6333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:41.598840 6333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:41.599586 6333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:41.599777 6333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:41.606639 6333 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:35821
I20260925 00:48:41.606645 6399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:35821 every 8 connection(s)
I20260925 00:48:41.607545 6333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:41.608409 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 6333
I20260925 00:48:41.608547 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:41.608808 997 external_mini_cluster.cc:1428] Setting key ad967a627d4a5693ce00f6b3d11e1820
I20260925 00:48:41.611060 6400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:41.616273 6400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68: Bootstrap starting.
I20260925 00:48:41.617843 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:41.610144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.0.249.126:35821)
0925 00:48:41.610761 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:41.610775 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:41.610809 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:48:41.611543 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:41.611561 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:41.611954 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:41.612286 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:41.612695 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.613699 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.614575 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.615053 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.615216 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.616085 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:41.616089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:41.616101 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:41.616512 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:41.616564 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:41.616585 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:41.616689 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:41.616837 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 00:48:41.617095 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":128,"threads_started":1}
I20260925 00:48:41.618476 6400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:41.619321 6400 log.cc:824] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:41.620699 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:48:41.621584 6400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68: No bootstrap required, opened a new log
I20260925 00:48:41.624974 6400 raft_consensus.cc:348] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddff4588ba54d99be967a29766a2d68" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35821 } }
I20260925 00:48:41.625238 6400 raft_consensus.cc:374] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:41.625319 6400 raft_consensus.cc:729] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddff4588ba54d99be967a29766a2d68, State: Initialized, Role: FOLLOWER
I20260925 00:48:41.625777 6400 consensus_queue.cc:260] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddff4588ba54d99be967a29766a2d68" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35821 } }
I20260925 00:48:41.625927 6400 raft_consensus.cc:388] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:41.626024 6400 raft_consensus.cc:482] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:41.626119 6400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:41.626965 6400 raft_consensus.cc:504] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddff4588ba54d99be967a29766a2d68" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35821 } }
I20260925 00:48:41.627338 6400 leader_election.cc:302] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fddff4588ba54d99be967a29766a2d68; no voters:
I20260925 00:48:41.627681 6400 leader_election.cc:288] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:41.627804 6405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:41.628055 6405 raft_consensus.cc:686] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [term 1 LEADER]: Becoming Leader. State: Replica: fddff4588ba54d99be967a29766a2d68, State: Running, Role: LEADER
I20260925 00:48:41.628337 6405 consensus_queue.cc:237] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddff4588ba54d99be967a29766a2d68" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35821 } }
I20260925 00:48:41.628829 6400 sys_catalog.cc:564] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:41.629951 6407 sys_catalog.cc:455] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fddff4588ba54d99be967a29766a2d68. Latest consensus state: current_term: 1 leader_uuid: "fddff4588ba54d99be967a29766a2d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddff4588ba54d99be967a29766a2d68" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35821 } } }
I20260925 00:48:41.630084 6407 sys_catalog.cc:458] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:41.630349 6406 sys_catalog.cc:455] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fddff4588ba54d99be967a29766a2d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddff4588ba54d99be967a29766a2d68" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35821 } } }
I20260925 00:48:41.630463 6406 sys_catalog.cc:458] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:41.630532 6414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:41.633476 6414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:41.638469 6414 catalog_manager.cc:1347] Generated new cluster ID: 6c1ae3a5963a470d89a2ce0839f8726b
I20260925 00:48:41.638530 6414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:41.649327 6414 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:41.650233 6414 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:41.656206 6414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fddff4588ba54d99be967a29766a2d68: Generated new TSK 0
I20260925 00:48:41.656915 6414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:48:41.680706 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:41.675393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.0.249.126:35821)
0925 00:48:41.675525 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:41.675530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:41.675692 (+ 162us) server_negotiation.cc:407] Connection header received
0925 00:48:41.675814 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:41.675817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:41.675870 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:41.675981 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0925 00:48:41.676299 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.677088 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.678132 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.678468 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.678585 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.678855 (+ 270us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 00:48:41.680349 (+ 1494us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 00:48:41.680570 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60854: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 00:48:41.680778 6403 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60854: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 00:48:41.682765 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:41.675273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:35821 (local address 127.0.0.1:60854)
0925 00:48:41.675610 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:41.675623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:41.675715 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:41.676052 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:41.676056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:41.676065 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 00:48:41.676174 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:41.676180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.677220 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.677249 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:41.678013 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:41.678022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.678602 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.678605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:41.678674 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.678696 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 00:48:41.680511 (+ 1815us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 00:48:41.682532 (+ 2021us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.249.126:35821: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":1676,"thread_start_us":108,"threads_started":1}
I20260925 00:48:41.683406 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 6333
E20260925 00:48:41.693753 6430 webserver.cc:533] Webserver: sslize failed:
E20260925 00:48:41.693815 6430 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T00:48:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (299 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T00:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:41Z Disabled control of system clock
I20260925 00:48:41.731674 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:38247
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:38247
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/testchainca.cert with env {}
W20260925 00:48:41.839213 6434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:41.839462 6434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:41.839502 6434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:41.843439 6434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:41.843494 6434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:41.843510 6434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:41.843526 6434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:41.843541 6434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 00:48:41.843562 6434 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 00:48:41.843580 6434 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 00:48:41.848659 6434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:38247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:38247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:41.849632 6434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:41.850546 6434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:41.858819 6439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:41.858903 6440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:41.858932 6434 server_base.cc:1034] running on GCE node
W20260925 00:48:41.858876 6442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:41.859602 6434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:41.860636 6434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:41.861790 6434 hybrid_clock.cc:648] HybridClock initialized: now 1790297321861761 us; error 43 us; skew 500 ppm
I20260925 00:48:41.863615 6434 webserver.cc:460] Webserver started at http://127.0.249.126:36451/ using document root <none> and password file <none>
I20260925 00:48:41.864130 6434 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:41.864187 6434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:41.864336 6434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:41.866338 6434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "03f2ea0172c54990b1ca31d0d5cce7c0"
format_stamp: "Formatted at 2026-09-25 00:48:41 on dist-test-slave-05m0"
server_key: "a83854d3a8cbc0ca0fa6353894a8b0fd"
server_key_iv: "89e0c5a6e8b60f57146be447f6eacc73"
server_key_version: "encryptionkey@0"
I20260925 00:48:41.866830 6434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "03f2ea0172c54990b1ca31d0d5cce7c0"
format_stamp: "Formatted at 2026-09-25 00:48:41 on dist-test-slave-05m0"
server_key: "a83854d3a8cbc0ca0fa6353894a8b0fd"
server_key_iv: "89e0c5a6e8b60f57146be447f6eacc73"
server_key_version: "encryptionkey@0"
I20260925 00:48:41.870335 6434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:48:41.872587 6448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:41.873868 6434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:41.874027 6434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "03f2ea0172c54990b1ca31d0d5cce7c0"
format_stamp: "Formatted at 2026-09-25 00:48:41 on dist-test-slave-05m0"
server_key: "a83854d3a8cbc0ca0fa6353894a8b0fd"
server_key_iv: "89e0c5a6e8b60f57146be447f6eacc73"
server_key_version: "encryptionkey@0"
I20260925 00:48:41.874143 6434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:41.883229 6434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:41.883848 6434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:41.884042 6434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:41.890870 6434 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:38247
I20260925 00:48:41.890882 6500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:38247 every 8 connection(s)
I20260925 00:48:41.891763 6434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:41.894707 6501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:41.896798 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 6434
I20260925 00:48:41.896930 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:41.897207 997 external_mini_cluster.cc:1428] Setting key 82127ef982e1eae0258c1f12be829ad7
I20260925 00:48:41.900494 6501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0: Bootstrap starting.
I20260925 00:48:41.902647 6501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:41.903362 6501 log.cc:824] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:41.905108 6501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0: No bootstrap required, opened a new log
I20260925 00:48:41.907244 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:41.898745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.0.249.126:38247)
0925 00:48:41.899420 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:41.899440 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:41.899478 (+ 38us) server_negotiation.cc:407] Connection header received
0925 00:48:41.900144 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:41.900162 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:41.900572 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:41.900991 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:41.901556 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.902659 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.903764 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.904288 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.904516 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.905586 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:41.905591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:41.905604 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:48:41.905967 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:41.906019 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:48:41.906039 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:41.906121 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:41.906285 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 00:48:41.906585 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":124,"threads_started":1}
I20260925 00:48:41.909046 6501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f2ea0172c54990b1ca31d0d5cce7c0" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38247 } }
I20260925 00:48:41.909312 6501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:41.909371 6501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f2ea0172c54990b1ca31d0d5cce7c0, State: Initialized, Role: FOLLOWER
I20260925 00:48:41.909878 6501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f2ea0172c54990b1ca31d0d5cce7c0" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38247 } }
I20260925 00:48:41.910045 6501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:41.910135 6501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:41.910233 6501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:41.911054 6501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f2ea0172c54990b1ca31d0d5cce7c0" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38247 } }
I20260925 00:48:41.911373 6501 leader_election.cc:302] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03f2ea0172c54990b1ca31d0d5cce7c0; no voters:
I20260925 00:48:41.911675 6501 leader_election.cc:288] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:41.911744 6506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:41.911926 6506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [term 1 LEADER]: Becoming Leader. State: Replica: 03f2ea0172c54990b1ca31d0d5cce7c0, State: Running, Role: LEADER
I20260925 00:48:41.912261 6506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f2ea0172c54990b1ca31d0d5cce7c0" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38247 } }
I20260925 00:48:41.912973 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:48:41.913370 6501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:41.913359 6506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03f2ea0172c54990b1ca31d0d5cce7c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f2ea0172c54990b1ca31d0d5cce7c0" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38247 } } }
I20260925 00:48:41.913529 6506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:41.913878 6510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:41.915653 6507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03f2ea0172c54990b1ca31d0d5cce7c0. Latest consensus state: current_term: 1 leader_uuid: "03f2ea0172c54990b1ca31d0d5cce7c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f2ea0172c54990b1ca31d0d5cce7c0" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 38247 } } }
I20260925 00:48:41.915822 6507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:41.916985 6510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:41.923026 6510 catalog_manager.cc:1347] Generated new cluster ID: 4b5d09af85394b6eacb75818cd8044cc
I20260925 00:48:41.923113 6510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:41.931228 6510 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:41.932144 6510 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:41.939163 6510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03f2ea0172c54990b1ca31d0d5cce7c0: Generated new TSK 0
I20260925 00:48:41.939827 6510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:48:42.008446 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:41.950784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.0.249.126:38247)
0925 00:48:41.950903 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:41.950918 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:41.951088 (+ 170us) server_negotiation.cc:407] Connection header received
0925 00:48:41.951281 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:41.951284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:41.951329 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:41.951445 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0925 00:48:41.951704 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.952242 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.953128 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.953360 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.953522 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.953842 (+ 320us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 00:48:42.007948 (+ 54106us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 00:48:42.008288 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52074: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 00:48:42.008544 6504 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52074: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 00:48:42.008623 6530 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:41.950722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:38247 (local address 127.0.0.1:52074)
0925 00:48:41.951001 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:41.951013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:41.951146 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:41.951462 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:41.951465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:41.951474 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 00:48:41.951596 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:41.951601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.952359 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.952363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:41.953001 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:41.953009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:41.953546 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:41.953550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:41.953642 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:41.953664 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 00:48:42.008194 (+ 54530us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 00:48:42.008392 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.249.126:38247: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
I20260925 00:48:42.009279 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 6434
2026-09-25T00:48:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (303 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:42 dist-test-slave-05m0 krb5kdc[6542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:42 dist-test-slave-05m0 krb5kdc[6542](info): set up 2 sockets
Sep 25 00:48:42 dist-test-slave-05m0 krb5kdc[6542](info): commencing operation
krb5kdc: starting...
W20260925 00:48:43.297883 997 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:43 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:43.418562 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36245
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:43.536502 6555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:43.536808 6555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:43.536890 6555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:43.541107 6555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:43.541193 6555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:43.541232 6555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:43.541292 6555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:43.541332 6555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:43.546674 6555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:43.547729 6555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:43.548739 6555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:43.557204 6563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:43.557204 6561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:43.557363 6555 server_base.cc:1034] running on GCE node
W20260925 00:48:43.557613 6560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:43.558143 6555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:43.559206 6555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:43.560388 6555 hybrid_clock.cc:648] HybridClock initialized: now 1790297323560355 us; error 45 us; skew 500 ppm
Sep 25 00:48:43 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:43.568264 6555 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:43.569298 6555 webserver.cc:460] Webserver started at http://127.0.249.126:33847/ using document root <none> and password file <none>
I20260925 00:48:43.569947 6555 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:43.570047 6555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:43.570297 6555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:43.572456 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "20e6feab4f2241aa89e184dfd66b9621"
format_stamp: "Formatted at 2026-09-25 00:48:43 on dist-test-slave-05m0"
server_key: "0b32ccebb8331f642150d5415a6d5864"
server_key_iv: "213c75dea0c86e1bb5a96d96b32c5fd1"
server_key_version: "encryptionkey@0"
I20260925 00:48:43.573087 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "20e6feab4f2241aa89e184dfd66b9621"
format_stamp: "Formatted at 2026-09-25 00:48:43 on dist-test-slave-05m0"
server_key: "0b32ccebb8331f642150d5415a6d5864"
server_key_iv: "213c75dea0c86e1bb5a96d96b32c5fd1"
server_key_version: "encryptionkey@0"
I20260925 00:48:43.577443 6555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 00:48:43.580188 6570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:43.581526 6555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:43.581666 6555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "20e6feab4f2241aa89e184dfd66b9621"
format_stamp: "Formatted at 2026-09-25 00:48:43 on dist-test-slave-05m0"
server_key: "0b32ccebb8331f642150d5415a6d5864"
server_key_iv: "213c75dea0c86e1bb5a96d96b32c5fd1"
server_key_version: "encryptionkey@0"
I20260925 00:48:43.581795 6555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:43.617168 6555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:43.620293 6555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:43.620519 6555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:43.628060 6555 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:36245
I20260925 00:48:43.628351 6622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:36245 every 8 connection(s)
I20260925 00:48:43.629098 6555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:43.631830 6623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:43.634696 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 6555
I20260925 00:48:43.634838 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:43.635124 997 external_mini_cluster.cc:1428] Setting key 2118e6c19219354e0b7aff6b7047724e
I20260925 00:48:43.637576 6623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621: Bootstrap starting.
I20260925 00:48:43.640172 6623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:43.640974 6623 log.cc:824] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:43.643105 6623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621: No bootstrap required, opened a new log
Sep 25 00:48:43 dist-test-slave-05m0 krb5kdc[6542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:43.647431 6623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e6feab4f2241aa89e184dfd66b9621" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36245 } }
I20260925 00:48:43.647696 6623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:43.647765 6623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e6feab4f2241aa89e184dfd66b9621, State: Initialized, Role: FOLLOWER
I20260925 00:48:43.648218 6623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e6feab4f2241aa89e184dfd66b9621" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36245 } }
I20260925 00:48:43.648363 6623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:43.648427 6623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:43.648521 6623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:43.649354 6623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e6feab4f2241aa89e184dfd66b9621" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36245 } }
I20260925 00:48:43.649760 6623 leader_election.cc:302] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e6feab4f2241aa89e184dfd66b9621; no voters:
I20260925 00:48:43.650082 6623 leader_election.cc:288] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:43.650174 6628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:43.650367 6628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [term 1 LEADER]: Becoming Leader. State: Replica: 20e6feab4f2241aa89e184dfd66b9621, State: Running, Role: LEADER
I20260925 00:48:43.650595 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:43.636568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.0.249.126:36245)
0925 00:48:43.636984 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:43.637007 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:43.637049 (+ 42us) server_negotiation.cc:407] Connection header received
0925 00:48:43.637905 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:43.637929 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:43.638317 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:43.638683 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:43.639367 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:43.640848 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:43.641857 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:43.642504 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:43.642679 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:43.645468 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:43.645475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:43.645489 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:43.645528 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:43.647533 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:43.648064 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:43.648069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:43.648073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:43.648181 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:43.648478 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:43.648484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:43.648486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:43.648855 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:43.649086 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:43.649345 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 00:48:43.649665 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20260925 00:48:43.650892 6628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e6feab4f2241aa89e184dfd66b9621" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36245 } }
I20260925 00:48:43.651281 6623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:43.652515 6630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e6feab4f2241aa89e184dfd66b9621. Latest consensus state: current_term: 1 leader_uuid: "20e6feab4f2241aa89e184dfd66b9621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e6feab4f2241aa89e184dfd66b9621" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36245 } } }
I20260925 00:48:43.652869 6630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:43.652779 6629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20e6feab4f2241aa89e184dfd66b9621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e6feab4f2241aa89e184dfd66b9621" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36245 } } }
I20260925 00:48:43.653223 6629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:43.653312 6641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:43.655959 6641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:43.661355 6641 catalog_manager.cc:1347] Generated new cluster ID: 02127839d738458691b2fb6c78d6aa94
I20260925 00:48:43.661494 6641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:43.672144 6641 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:43.673177 6641 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:43.680691 6641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20e6feab4f2241aa89e184dfd66b9621: Generated new TSK 0
I20260925 00:48:43.681330 6641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:48:43.712387 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36245
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:43.821177 6649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:43.821476 6649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:43.821584 6649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:43.825515 6649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:43.825563 6649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:43.825650 6649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:43.830956 6649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:43.831929 6649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:43.832945 6649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:43.842312 6655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:43.842278 6654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:43.842288 6657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:43.842459 6649 server_base.cc:1034] running on GCE node
I20260925 00:48:43.842924 6649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:43.843585 6649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:43.844771 6649 hybrid_clock.cc:648] HybridClock initialized: now 1790297323844747 us; error 38 us; skew 500 ppm
Sep 25 00:48:43 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:43.847496 6649 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:43.848380 6649 webserver.cc:460] Webserver started at http://127.0.249.65:41835/ using document root <none> and password file <none>
I20260925 00:48:43.848919 6649 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:43.849007 6649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:43.849207 6649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:43.851236 6649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "97ac984b35524df4b8d2c472fa64566a"
format_stamp: "Formatted at 2026-09-25 00:48:43 on dist-test-slave-05m0"
server_key: "28d1e8d4b3af95390eb8a47e5d8c31b9"
server_key_iv: "5f5cf7e4ed91bc8d2243ad21b584b6e3"
server_key_version: "encryptionkey@0"
I20260925 00:48:43.851778 6649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "97ac984b35524df4b8d2c472fa64566a"
format_stamp: "Formatted at 2026-09-25 00:48:43 on dist-test-slave-05m0"
server_key: "28d1e8d4b3af95390eb8a47e5d8c31b9"
server_key_iv: "5f5cf7e4ed91bc8d2243ad21b584b6e3"
server_key_version: "encryptionkey@0"
I20260925 00:48:43.855535 6649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 00:48:43.857895 6664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:43.859033 6649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:43.859160 6649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "97ac984b35524df4b8d2c472fa64566a"
format_stamp: "Formatted at 2026-09-25 00:48:43 on dist-test-slave-05m0"
server_key: "28d1e8d4b3af95390eb8a47e5d8c31b9"
server_key_iv: "5f5cf7e4ed91bc8d2243ad21b584b6e3"
server_key_version: "encryptionkey@0"
I20260925 00:48:43.859265 6649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:43.889727 6649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:43.893358 6649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:43.893591 6649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:43.894156 6649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:43.894942 6649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:43.895009 6649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:43.895071 6649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:43.895121 6649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:43.903209 6649 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:39135
I20260925 00:48:43.903283 6777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:39135 every 8 connection(s)
I20260925 00:48:43.904094 6649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:43.908604 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 6649
I20260925 00:48:43.908720 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:43.908975 997 external_mini_cluster.cc:1428] Setting key 02fbc2fe9985bf1324928e5477a61b93
Sep 25 00:48:43 dist-test-slave-05m0 krb5kdc[6542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:43.918109 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:43.905859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:34335 (local address 127.0.249.126:36245)
0925 00:48:43.905991 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:43.905995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:43.906789 (+ 794us) server_negotiation.cc:407] Connection header received
0925 00:48:43.908228 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:43.908231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:43.908284 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:43.908415 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:43.909759 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:43.910386 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:43.911494 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:43.911730 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:43.911840 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:43.914736 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:43.914740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:43.914744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:43.914771 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:43.916166 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:43.916652 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:43.916656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:43.916658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:43.916733 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:43.917055 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:43.917058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:43.917060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:43.917221 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:43.917322 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:43.917826 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 00:48:43.917942 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:48:43.919001 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:43.906224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36245 (local address 127.0.249.65:34335)
0925 00:48:43.906651 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:43.906691 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:43.908027 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:43.908580 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:43.908586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:43.909033 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:43.909618 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:43.909634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:43.910507 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:43.910510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:43.911371 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:43.911380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:43.911871 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:43.911875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:43.912111 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:43.912718 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:43.912742 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:43.914581 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:43.916277 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:43.916283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:43.916296 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:43.916555 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:43.916827 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:43.916830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:43.916832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:43.916958 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:43.917320 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:43.917327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:43.917590 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 00:48:43.917773 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 00:48:43.918058 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":126,"threads_started":1}
I20260925 00:48:43.920183 6778 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36245
I20260925 00:48:43.920424 6778 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
I20260925 00:48:43.920985 6778 heartbeater.cc:511] Master 127.0.249.126:36245 requested a full tablet report, sending...
I20260925 00:48:43.922487 6587 ts_manager.cc:194] Registered new tserver with Master: 97ac984b35524df4b8d2c472fa64566a (127.0.249.65:39135)
I20260925 00:48:43.923557 6587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:34335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:43.934770 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36245
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:44.045233 6783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:44.045493 6783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:44.045536 6783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:44.049424 6783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:44.049470 6783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:44.049572 6783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:44.054888 6783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:44.055779 6783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:44.056711 6783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:44.066035 6791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:44.066121 6789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:44.066075 6788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:44.066279 6783 server_base.cc:1034] running on GCE node
I20260925 00:48:44.066604 6783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:44.067183 6783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:44.068339 6783 hybrid_clock.cc:648] HybridClock initialized: now 1790297324068323 us; error 27 us; skew 500 ppm
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:44.071084 6783 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:44.071888 6783 webserver.cc:460] Webserver started at http://127.0.249.66:39863/ using document root <none> and password file <none>
I20260925 00:48:44.072381 6783 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:44.072444 6783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:44.072597 6783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:44.074658 6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "d39d4ef155cc4d489e1d43577632cde3"
format_stamp: "Formatted at 2026-09-25 00:48:44 on dist-test-slave-05m0"
server_key: "2ccc1f76cddb7facc47767c49eaa7fe5"
server_key_iv: "f2a0df10d3f8120fcf905865c9b2dfbc"
server_key_version: "encryptionkey@0"
I20260925 00:48:44.075170 6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "d39d4ef155cc4d489e1d43577632cde3"
format_stamp: "Formatted at 2026-09-25 00:48:44 on dist-test-slave-05m0"
server_key: "2ccc1f76cddb7facc47767c49eaa7fe5"
server_key_iv: "f2a0df10d3f8120fcf905865c9b2dfbc"
server_key_version: "encryptionkey@0"
I20260925 00:48:44.078630 6783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:48:44.080910 6798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:44.082099 6783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:44.082193 6783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "d39d4ef155cc4d489e1d43577632cde3"
format_stamp: "Formatted at 2026-09-25 00:48:44 on dist-test-slave-05m0"
server_key: "2ccc1f76cddb7facc47767c49eaa7fe5"
server_key_iv: "f2a0df10d3f8120fcf905865c9b2dfbc"
server_key_version: "encryptionkey@0"
I20260925 00:48:44.082271 6783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:44.102625 6783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:44.106149 6783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:44.106300 6783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:44.106801 6783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:44.107538 6783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:44.107585 6783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:44.107625 6783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:44.107645 6783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:44.116343 6783 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:44027
I20260925 00:48:44.116367 6911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:44027 every 8 connection(s)
I20260925 00:48:44.117169 6783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:44.120730 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 6783
I20260925 00:48:44.120821 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:44.121138 997 external_mini_cluster.cc:1428] Setting key 06e6355ce7f15586ee5d4deeb48055cf
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:48:44.131309 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:44.118898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:47849 (local address 127.0.249.126:36245)
0925 00:48:44.119043 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:44.119046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:44.119773 (+ 727us) server_negotiation.cc:407] Connection header received
0925 00:48:44.121203 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:44.121206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:44.121262 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:44.121369 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:44.122809 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.123317 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.124717 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.124943 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.125027 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:44.127766 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:44.127769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:44.127771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:44.127794 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:44.129250 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.129789 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.129794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.129795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.129874 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.130219 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.130222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.130224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.130446 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:44.130547 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:44.131019 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 00:48:44.131140 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:48:44.132026 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:44.119221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36245 (local address 127.0.249.66:47849)
0925 00:48:44.119630 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:44.119669 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:44.120987 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:44.121575 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:44.121584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:44.121994 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:44.122629 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:44.122651 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.123447 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.123450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:44.124584 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:44.124598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.125095 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.125101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:44.125310 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:44.125919 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:44.125942 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:44.127598 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:44.129375 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:44.129381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:44.129423 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:44.129682 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:44.129981 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:44.129984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:44.129986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:44.130117 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:44.130545 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:44.130552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:44.130784 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 00:48:44.130962 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 00:48:44.131259 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
I20260925 00:48:44.133188 6912 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36245
I20260925 00:48:44.133441 6912 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:44.133985 6912 heartbeater.cc:511] Master 127.0.249.126:36245 requested a full tablet report, sending...
I20260925 00:48:44.135039 6587 ts_manager.cc:194] Registered new tserver with Master: d39d4ef155cc4d489e1d43577632cde3 (127.0.249.66:44027)
I20260925 00:48:44.135583 6587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:47849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:44.148551 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36245
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:44.258770 6917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:44.259090 6917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:44.259181 6917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:44.263899 6917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:44.263989 6917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:44.264091 6917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:44.269747 6917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:44.270838 6917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:44.271957 6917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:44.281901 6923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:44.281909 6922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:44.281936 6925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:44.282330 6917 server_base.cc:1034] running on GCE node
I20260925 00:48:44.282711 6917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:44.283334 6917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:44.284513 6917 hybrid_clock.cc:648] HybridClock initialized: now 1790297324284500 us; error 24 us; skew 500 ppm
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:44.287331 6917 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:44.288169 6917 webserver.cc:460] Webserver started at http://127.0.249.67:39531/ using document root <none> and password file <none>
I20260925 00:48:44.288709 6917 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:44.288792 6917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:44.289004 6917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:44.290992 6917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "3619eaa70db0476b8c2de7e9d6ff18bc"
format_stamp: "Formatted at 2026-09-25 00:48:44 on dist-test-slave-05m0"
server_key: "769fd939ab6e49799262b1db656faa1b"
server_key_iv: "a3aeeed159301b6af96593365b09c0c4"
server_key_version: "encryptionkey@0"
I20260925 00:48:44.291527 6917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "3619eaa70db0476b8c2de7e9d6ff18bc"
format_stamp: "Formatted at 2026-09-25 00:48:44 on dist-test-slave-05m0"
server_key: "769fd939ab6e49799262b1db656faa1b"
server_key_iv: "a3aeeed159301b6af96593365b09c0c4"
server_key_version: "encryptionkey@0"
I20260925 00:48:44.295113 6917 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 00:48:44.297557 6932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:44.298776 6917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:44.298902 6917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "3619eaa70db0476b8c2de7e9d6ff18bc"
format_stamp: "Formatted at 2026-09-25 00:48:44 on dist-test-slave-05m0"
server_key: "769fd939ab6e49799262b1db656faa1b"
server_key_iv: "a3aeeed159301b6af96593365b09c0c4"
server_key_version: "encryptionkey@0"
I20260925 00:48:44.299009 6917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:44.315757 6917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:44.318907 6917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:44.319070 6917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:44.319551 6917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:44.320242 6917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:44.320286 6917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:44.320322 6917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:44.320343 6917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:44.328670 6917 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:41711
I20260925 00:48:44.328753 7045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:41711 every 8 connection(s)
I20260925 00:48:44.329591 6917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:44.334158 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 6917
I20260925 00:48:44.334273 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:44.334510 997 external_mini_cluster.cc:1428] Setting key 5cb5f31381446353b8489bf14f458031
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:44.343016 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:44.331286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:49347 (local address 127.0.249.126:36245)
0925 00:48:44.331435 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:44.331439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:44.332182 (+ 743us) server_negotiation.cc:407] Connection header received
0925 00:48:44.333624 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:44.333628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:44.333684 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:44.333789 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:44.335090 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.335880 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.337116 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.337327 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.337460 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:44.339757 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:44.339760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:44.339762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:44.339784 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:44.341215 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.341804 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.341807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.341808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.341882 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.342146 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.342148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.342149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.342304 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:44.342372 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:44.342805 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 00:48:44.342900 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 00:48:44.343750 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:44.331626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36245 (local address 127.0.249.67:49347)
0925 00:48:44.332042 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:44.332082 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:44.333350 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:44.333978 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:44.333985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:44.334362 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:44.334958 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:44.334981 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.335989 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.335991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:44.337002 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:44.337012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.337495 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.337497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:44.337633 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:44.338209 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:44.338224 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:44.339614 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:44.341307 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:44.341311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:44.341322 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:44.341711 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:44.341962 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:44.341965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:44.341966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:44.342086 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:44.342381 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:44.342386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:44.342603 (+ 217us) client_negotiation.cc:769] Sending connection context
0925 00:48:44.342742 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 00:48:44.342967 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260925 00:48:44.344849 7046 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36245
I20260925 00:48:44.345077 7046 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:44.345584 7046 heartbeater.cc:511] Master 127.0.249.126:36245 requested a full tablet report, sending...
I20260925 00:48:44.346516 6587 ts_manager.cc:194] Registered new tserver with Master: 3619eaa70db0476b8c2de7e9d6ff18bc (127.0.249.67:41711)
I20260925 00:48:44.347074 6587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:49347
I20260925 00:48:44.348174 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/test-admin.keytab.
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:44.363277 997 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 00:48:44.365964 7053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:44.364588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.0.249.65:39135)
0925 00:48:44.364906 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:44.364919 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:44.364943 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:48:44.364989 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:44.364994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:44.365175 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:44.365292 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:44.365796 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39962: BlockingRecv error: recv got EOF from 127.0.0.1:39962 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":122,"threads_started":1}
W20260925 00:48:44.365996 997 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5cc' of type FILE
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:44.366972 997 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:44.367841 997 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:44.376861 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:44.368637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.0.249.65:39135)
0925 00:48:44.368753 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:44.368757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:44.368854 (+ 97us) server_negotiation.cc:407] Connection header received
0925 00:48:44.368961 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:44.368963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:44.369001 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:44.369072 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:44.369594 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.370348 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.371131 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.371467 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.371545 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:44.373716 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:44.373726 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:44.373745 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:44.373767 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:44.375431 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.375798 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.375803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.375804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.375875 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.376103 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.376105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.376106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.376321 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:44.376462 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:44.376618 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 00:48:44.376731 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 00:48:44.380677 7053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:44.379660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.0.249.65:39135)
0925 00:48:44.379787 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:44.379791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:44.379889 (+ 98us) server_negotiation.cc:407] Connection header received
0925 00:48:44.380053 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:44.380055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:44.380093 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:44.380181 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:44.380561 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39984: BlockingRecv error: recv got EOF from 127.0.0.1:39984 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 00:48:44.380821 997 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5cc' of type FILE
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:44.381821 997 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:44.382663 997 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:44.391072 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:44.383385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.0.249.65:39135)
0925 00:48:44.383494 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:44.383497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:44.383616 (+ 119us) server_negotiation.cc:407] Connection header received
0925 00:48:44.383698 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:44.383700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:44.383731 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:44.383793 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:44.384223 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.384734 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.385616 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.385829 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.385904 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:44.388153 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:44.388157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:44.388159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:44.388180 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:44.389499 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.389911 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.389915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.389917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.389997 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.390273 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.390277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.390279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.390440 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:44.390573 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:44.390817 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 00:48:44.390939 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 00:48:44.393680 7053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:44.392784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.0.249.65:39135)
0925 00:48:44.392926 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:44.392930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:44.392967 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:48:44.393083 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:44.393085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:44.393118 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:44.393182 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:44.393589 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39994: BlockingRecv error: recv got EOF from 127.0.0.1:39994 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 00:48:44.393889 997 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790297292685301-997-0/krb5kdc/krb5cc' of type FILE
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:44.394899 997 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:44.395812 997 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[6542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:48:44.404635 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:44.396566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.0.249.65:39135)
0925 00:48:44.396676 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:44.396680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:44.396818 (+ 138us) server_negotiation.cc:407] Connection header received
0925 00:48:44.396967 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:44.396970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:44.397013 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:44.397091 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:44.397608 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.398101 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.398886 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:44.399111 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:44.399237 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:44.401615 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:44.401619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:44.401624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:44.401649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:44.403086 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.403498 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.403503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.403505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.403581 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:44.403851 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:44.403856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:44.403857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:44.404019 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:44.404165 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:44.404384 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 00:48:44.404486 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:48:44.405704 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 6649
I20260925 00:48:44.411684 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 6783
I20260925 00:48:44.416797 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 6917
I20260925 00:48:44.421844 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 6555
2026-09-25T00:48:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2170 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[7066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[7066](info): set up 2 sockets
Sep 25 00:48:44 dist-test-slave-05m0 krb5kdc[7066](info): commencing operation
krb5kdc: starting...
W20260925 00:48:45.473832 997 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:45 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.126.keytab.
Entry for principal oryx/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.126.keytab.
I20260925 00:48:45.592639 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46365
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.126.keytab
--principal=oryx/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:45.698925 7079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:45.699221 7079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:45.699301 7079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:45.703220 7079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:45.703305 7079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:45.703342 7079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:45.703400 7079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:45.703439 7079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:45.708487 7079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.126.keytab
--principal=oryx/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:45.709542 7079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:45.710520 7079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:45.718533 7085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:45.718573 7084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:45.718581 7087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:45.718892 7079 server_base.cc:1034] running on GCE node
I20260925 00:48:45.719395 7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:45.720417 7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:45.721570 7079 hybrid_clock.cc:648] HybridClock initialized: now 1790297325721553 us; error 29 us; skew 500 ppm
Sep 25 00:48:45 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:45.724117 7079 init.cc:377] Logged in from keytab as oryx/127.0.249.126@KRBTEST.COM (short username oryx)
I20260925 00:48:45.724974 7079 webserver.cc:460] Webserver started at http://127.0.249.126:38329/ using document root <none> and password file <none>
I20260925 00:48:45.725530 7079 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:45.725595 7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:45.725756 7079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:45.727634 7079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "c980451b649c44b78e24958553b2e521"
format_stamp: "Formatted at 2026-09-25 00:48:45 on dist-test-slave-05m0"
server_key: "53aee69a1d74bc3dec554d1e5581edd0"
server_key_iv: "7acaa7a2a76ada8661a5330d7e87dc6b"
server_key_version: "encryptionkey@0"
I20260925 00:48:45.728155 7079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "c980451b649c44b78e24958553b2e521"
format_stamp: "Formatted at 2026-09-25 00:48:45 on dist-test-slave-05m0"
server_key: "53aee69a1d74bc3dec554d1e5581edd0"
server_key_iv: "7acaa7a2a76ada8661a5330d7e87dc6b"
server_key_version: "encryptionkey@0"
I20260925 00:48:45.731613 7079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:48:45.733884 7094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:45.735080 7079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:45.735206 7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "c980451b649c44b78e24958553b2e521"
format_stamp: "Formatted at 2026-09-25 00:48:45 on dist-test-slave-05m0"
server_key: "53aee69a1d74bc3dec554d1e5581edd0"
server_key_iv: "7acaa7a2a76ada8661a5330d7e87dc6b"
server_key_version: "encryptionkey@0"
I20260925 00:48:45.735304 7079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:45.752408 7079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:45.755362 7079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:45.755553 7079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:45.762248 7079 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:46365
I20260925 00:48:45.762545 7146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:46365 every 8 connection(s)
I20260925 00:48:45.763207 7079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:45.765894 7147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:45.768628 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 7079
I20260925 00:48:45.768766 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:45.769120 997 external_mini_cluster.cc:1428] Setting key 7984ccb0375e9617c67f67347fabc7fa
I20260925 00:48:45.772110 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: Bootstrap starting.
I20260925 00:48:45.774343 7147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:45.775054 7147 log.cc:824] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:45.777294 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: No bootstrap required, opened a new log
Sep 25 00:48:45 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:45.780485 7147 raft_consensus.cc:348] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } }
I20260925 00:48:45.780736 7147 raft_consensus.cc:374] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:45.780833 7147 raft_consensus.cc:729] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c980451b649c44b78e24958553b2e521, State: Initialized, Role: FOLLOWER
I20260925 00:48:45.781307 7147 consensus_queue.cc:260] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } }
I20260925 00:48:45.781524 7147 raft_consensus.cc:388] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:45.781606 7147 raft_consensus.cc:482] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:45.781715 7147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:45.782785 7147 raft_consensus.cc:504] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } }
I20260925 00:48:45.783154 7147 leader_election.cc:302] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c980451b649c44b78e24958553b2e521; no voters:
I20260925 00:48:45.783533 7147 leader_election.cc:288] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:45.783610 7152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:45.783823 7152 raft_consensus.cc:686] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 1 LEADER]: Becoming Leader. State: Replica: c980451b649c44b78e24958553b2e521, State: Running, Role: LEADER
I20260925 00:48:45.784175 7152 consensus_queue.cc:237] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } }
I20260925 00:48:45.784328 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:45.770498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.0.249.126:46365)
0925 00:48:45.770917 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:45.770931 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:45.770966 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:48:45.771572 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:45.771588 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:45.771907 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:45.772248 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:45.773053 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:45.774106 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:45.774985 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:45.775581 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:45.775765 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:45.778697 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:45.778702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:45.778713 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:45.778738 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:45.781126 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:45.781645 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:45.781651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:45.781663 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:45.781831 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:45.782142 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:45.782144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:45.782145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:45.782502 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:45.782648 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:45.783005 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 00:48:45.783343 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
I20260925 00:48:45.784828 7147 sys_catalog.cc:564] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:45.785422 7152 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } }
I20260925 00:48:45.785841 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c980451b649c44b78e24958553b2e521. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } }
I20260925 00:48:45.786072 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:45.786330 7152 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:45.786661 7161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:45.789767 7161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:45.795006 7161 catalog_manager.cc:1347] Generated new cluster ID: f3da4ca6cc444f349e8b2510b228bff8
I20260925 00:48:45.795073 7161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:45.803669 7161 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:45.804693 7161 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:45.814683 7161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: Generated new TSK 0
I20260925 00:48:45.815449 7161 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab.
Entry for principal oryx/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab.
I20260925 00:48:45.848765 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46365
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab
--principal=oryx/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:45.974553 7173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:45.974850 7173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:45.974892 7173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:45.978917 7173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:45.978963 7173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:45.979017 7173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:45.984102 7173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab
--principal=oryx/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:45.985124 7173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:45.986224 7173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:45.996381 7179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:45.996385 7178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:45.996659 7173 server_base.cc:1034] running on GCE node
W20260925 00:48:45.996811 7181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:45.997202 7173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:45.997854 7173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:45.999022 7173 hybrid_clock.cc:648] HybridClock initialized: now 1790297325998983 us; error 50 us; skew 500 ppm
Sep 25 00:48:45 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:46.001850 7173 init.cc:377] Logged in from keytab as oryx/127.0.249.65@KRBTEST.COM (short username oryx)
I20260925 00:48:46.002727 7173 webserver.cc:460] Webserver started at http://127.0.249.65:38379/ using document root <none> and password file <none>
I20260925 00:48:46.003306 7173 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:46.003402 7173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:46.003638 7173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:46.005723 7173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "1067f1241a5c4c69ba9acf81da15eed5"
format_stamp: "Formatted at 2026-09-25 00:48:45 on dist-test-slave-05m0"
server_key: "9a48f55cdae26d26838693a7ba819a08"
server_key_iv: "abe620ff210a95b79b8889e7dea4cb74"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.006289 7173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "1067f1241a5c4c69ba9acf81da15eed5"
format_stamp: "Formatted at 2026-09-25 00:48:45 on dist-test-slave-05m0"
server_key: "9a48f55cdae26d26838693a7ba819a08"
server_key_iv: "abe620ff210a95b79b8889e7dea4cb74"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.009840 7173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:46.012171 7188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.013469 7173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:46.013592 7173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "1067f1241a5c4c69ba9acf81da15eed5"
format_stamp: "Formatted at 2026-09-25 00:48:45 on dist-test-slave-05m0"
server_key: "9a48f55cdae26d26838693a7ba819a08"
server_key_iv: "abe620ff210a95b79b8889e7dea4cb74"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.013706 7173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:46.028141 7173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:46.031360 7173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:46.031567 7173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:46.032159 7173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:46.032970 7173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:46.033041 7173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.033119 7173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:46.033154 7173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.041855 7173 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:33341
I20260925 00:48:46.041875 7301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:33341 every 8 connection(s)
I20260925 00:48:46.042752 7173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:46.045800 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 7173
I20260925 00:48:46.045962 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:46.046340 997 external_mini_cluster.cc:1428] Setting key b062df76f0c8470ca9acb98d90abb022
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:48:46.057497 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.044479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:53929 (local address 127.0.249.126:46365)
0925 00:48:46.044657 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.044661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.045571 (+ 910us) server_negotiation.cc:407] Connection header received
0925 00:48:46.047228 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.047231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.047274 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.047346 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.048844 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.049442 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.050615 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.050851 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.050963 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.053800 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.053803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.053805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.053828 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.055466 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.056087 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.056090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.056091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.056165 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.056471 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.056473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.056474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.056636 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.056711 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.057199 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.057299 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 00:48:46.058303 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.044882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.65:53929)
0925 00:48:46.045339 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.045468 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.047005 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.047546 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.047555 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.048004 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.048687 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.048709 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.049579 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.049583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.050455 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.050464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.051002 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.051006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.051186 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.051776 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.051792 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.053637 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.055590 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.055595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.055609 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.055976 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.056252 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.056254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.056256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.056385 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.056724 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.056728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.056994 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.057143 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.057376 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.66@KRBTEST.COM" created.
I20260925 00:48:46.059618 7302 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46365
I20260925 00:48:46.059846 7302 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:46.060420 7302 heartbeater.cc:511] Master 127.0.249.126:46365 requested a full tablet report, sending...
I20260925 00:48:46.062031 7111 ts_manager.cc:194] Registered new tserver with Master: 1067f1241a5c4c69ba9acf81da15eed5 (127.0.249.65:33341)
I20260925 00:48:46.063133 7111 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:53929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab.
Entry for principal oryx/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab.
I20260925 00:48:46.072758 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46365
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab
--principal=oryx/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:46.181346 7307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:46.181676 7307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:46.181756 7307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:46.185885 7307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:46.185961 7307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:46.186046 7307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:46.191244 7307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab
--principal=oryx/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:46.192273 7307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:46.193516 7307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:46.202970 7313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:46.202979 7315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:46.203095 7307 server_base.cc:1034] running on GCE node
W20260925 00:48:46.202979 7312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:46.203606 7307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:46.204221 7307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:46.205438 7307 hybrid_clock.cc:648] HybridClock initialized: now 1790297326205416 us; error 29 us; skew 500 ppm
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:46.208086 7307 init.cc:377] Logged in from keytab as oryx/127.0.249.66@KRBTEST.COM (short username oryx)
I20260925 00:48:46.208914 7307 webserver.cc:460] Webserver started at http://127.0.249.66:34479/ using document root <none> and password file <none>
I20260925 00:48:46.209519 7307 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:46.209604 7307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:46.209820 7307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:46.211822 7307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "8cd5d1b8797840909a3870baf96efb98"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "5162ca5deffc6191153b82e03d76ca5b"
server_key_iv: "eff351e4312bd2b48a4400d57161dfd3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.212397 7307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "8cd5d1b8797840909a3870baf96efb98"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "5162ca5deffc6191153b82e03d76ca5b"
server_key_iv: "eff351e4312bd2b48a4400d57161dfd3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.216198 7307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 00:48:46.218684 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.219844 7307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:46.219977 7307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "8cd5d1b8797840909a3870baf96efb98"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "5162ca5deffc6191153b82e03d76ca5b"
server_key_iv: "eff351e4312bd2b48a4400d57161dfd3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.220084 7307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:46.239702 7307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:46.242695 7307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:46.242893 7307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:46.243492 7307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:46.244329 7307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:46.244396 7307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.244459 7307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:46.244499 7307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.253252 7307 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:45821
I20260925 00:48:46.253273 7435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:45821 every 8 connection(s)
I20260925 00:48:46.254215 7307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:46.258533 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 7307
I20260925 00:48:46.258662 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:46.258996 997 external_mini_cluster.cc:1428] Setting key 7b48e077c5d64bbb3f11a8ca175ce071
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:46.268368 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.256008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:40561 (local address 127.0.249.126:46365)
0925 00:48:46.256146 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.256155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.256890 (+ 735us) server_negotiation.cc:407] Connection header received
0925 00:48:46.258226 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.258229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.258274 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.258349 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.259818 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.260342 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.261588 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.261952 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.262072 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.264454 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.264460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.264467 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.264500 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.266291 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.266764 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.266767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.266769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.266860 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.267173 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.267178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.267181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.267433 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.267545 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.268004 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.268139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for oryx/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:48:46.269006 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.256309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.66:40561)
0925 00:48:46.256726 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.256768 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.258029 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.258592 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.258601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.259105 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.259666 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.259680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.260467 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.260470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.261367 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.261375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.262111 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.262114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.262257 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.262863 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.262878 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.264290 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.266401 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.266405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.266416 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.266668 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.266955 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.266957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.266958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.267083 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.267546 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.267551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.267757 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.267914 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.268146 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260925 00:48:46.270270 7436 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46365
I20260925 00:48:46.270511 7436 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:46.271029 7436 heartbeater.cc:511] Master 127.0.249.126:46365 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.67@KRBTEST.COM" created.
I20260925 00:48:46.272157 7111 ts_manager.cc:194] Registered new tserver with Master: 8cd5d1b8797840909a3870baf96efb98 (127.0.249.66:45821)
I20260925 00:48:46.272706 7111 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:40561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab.
Entry for principal oryx/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab.
I20260925 00:48:46.287876 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46365
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab
--principal=oryx/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:46.396124 7441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:46.396366 7441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:46.396406 7441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:46.400214 7441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:46.400257 7441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:46.400311 7441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:46.405337 7441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab
--principal=oryx/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:46.406266 7441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:46.407161 7441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:46.416307 7446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:46.416400 7447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:46.416321 7449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:46.416507 7441 server_base.cc:1034] running on GCE node
I20260925 00:48:46.417060 7441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:46.417747 7441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:46.418926 7441 hybrid_clock.cc:648] HybridClock initialized: now 1790297326418915 us; error 28 us; skew 500 ppm
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:46.421586 7441 init.cc:377] Logged in from keytab as oryx/127.0.249.67@KRBTEST.COM (short username oryx)
I20260925 00:48:46.422348 7441 webserver.cc:460] Webserver started at http://127.0.249.67:43557/ using document root <none> and password file <none>
I20260925 00:48:46.422888 7441 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:46.422962 7441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:46.423175 7441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:46.425140 7441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "0edc623c84004f16abb8d09a2cce0953"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "edaa1a04e404c5944a493ef0b2f8ba82"
server_key_iv: "d391fa02fc30f0559af080023ca80f64"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.425726 7441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "0edc623c84004f16abb8d09a2cce0953"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "edaa1a04e404c5944a493ef0b2f8ba82"
server_key_iv: "d391fa02fc30f0559af080023ca80f64"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.429147 7441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:46.431411 7456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.432503 7441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:46.432628 7441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "0edc623c84004f16abb8d09a2cce0953"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "edaa1a04e404c5944a493ef0b2f8ba82"
server_key_iv: "d391fa02fc30f0559af080023ca80f64"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.432722 7441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:46.443953 7441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:46.446705 7441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:46.446897 7441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:46.447435 7441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:46.448204 7441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:46.448249 7441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.448315 7441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:46.448354 7441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.456919 7441 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:34397
I20260925 00:48:46.456998 7569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:34397 every 8 connection(s)
I20260925 00:48:46.457850 7441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:46.463752 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 7441
I20260925 00:48:46.463869 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:46.464195 997 external_mini_cluster.cc:1428] Setting key c780302ece2eefbe606314da98d290a8
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:46.471220 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.459636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:55829 (local address 127.0.249.126:46365)
0925 00:48:46.459773 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.459777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.460566 (+ 789us) server_negotiation.cc:407] Connection header received
0925 00:48:46.462021 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.462025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.462082 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.462164 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.463294 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.464044 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.465622 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.465827 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.465908 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.468279 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.468281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.468283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.468306 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.469571 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.469990 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.469993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.469994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.470062 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.470348 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.470350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.470351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.470529 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.470601 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.471015 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.471106 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:48:46.472010 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.460008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:55829)
0925 00:48:46.460415 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.460456 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.461751 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.462328 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.462334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.462685 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.463147 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.463159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.464169 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.464172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.465490 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.465500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.465965 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.465969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.466182 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.466737 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.466753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.468142 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.469663 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.469667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.469677 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.469911 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.470136 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.470138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.470139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.470257 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.470611 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.470616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.470823 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.470970 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.471226 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260925 00:48:46.473047 7570 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46365
I20260925 00:48:46.473284 7570 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:46.473762 7570 heartbeater.cc:511] Master 127.0.249.126:46365 requested a full tablet report, sending...
I20260925 00:48:46.474742 7111 ts_manager.cc:194] Registered new tserver with Master: 0edc623c84004f16abb8d09a2cce0953 (127.0.249.67:34397)
I20260925 00:48:46.475250 7111 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:55829
I20260925 00:48:46.477864 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.249.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.125.keytab.
Entry for principal oryx/127.0.249.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.125.keytab.
I20260925 00:48:46.502629 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.125:36727
--webserver_interface=127.0.249.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.125.keytab
--principal=oryx/127.0.249.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46365,127.0.249.125:36727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:46.609869 7575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:46.610128 7575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:46.610224 7575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:46.614369 7575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:46.614439 7575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:46.614473 7575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:46.614535 7575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:46.614573 7575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:46.619783 7575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46365,127.0.249.125:36727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.125.keytab
--principal=oryx/127.0.249.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.125:36727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.249.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
W20260925 00:48:46.620009 7575 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.249.126:46365,127.0.249.125:36727'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 00:48:46.620822 7575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:46.621834 7575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:46.629705 7583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:46.629732 7575 server_base.cc:1034] running on GCE node
W20260925 00:48:46.629686 7580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:46.629688 7581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:46.630484 7575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:46.631474 7575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:46.632664 7575 hybrid_clock.cc:648] HybridClock initialized: now 1790297326632650 us; error 27 us; skew 500 ppm
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:46.635311 7575 init.cc:377] Logged in from keytab as oryx/127.0.249.125@KRBTEST.COM (short username oryx)
I20260925 00:48:46.636132 7575 webserver.cc:460] Webserver started at http://127.0.249.125:41801/ using document root <none> and password file <none>
I20260925 00:48:46.636682 7575 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:46.636739 7575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:46.636895 7575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:46.638888 7575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data/instance:
uuid: "46f8bacab3ba42be8f27d3ce89d5b46e"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "f3aa7ab16049e0d9cd406cc726edae8f"
server_key_iv: "fc1bdcdd0190f56ae9ea412d87bc33f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.639438 7575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal/instance:
uuid: "46f8bacab3ba42be8f27d3ce89d5b46e"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "f3aa7ab16049e0d9cd406cc726edae8f"
server_key_iv: "fc1bdcdd0190f56ae9ea412d87bc33f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.642774 7575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 00:48:46.644891 7590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.646067 7575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 00:48:46.646184 7575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal
uuid: "46f8bacab3ba42be8f27d3ce89d5b46e"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "f3aa7ab16049e0d9cd406cc726edae8f"
server_key_iv: "fc1bdcdd0190f56ae9ea412d87bc33f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.646278 7575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:46.663724 7575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:46.666630 7575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:46.666826 7575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:46.673727 7575 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.125:36727
I20260925 00:48:46.674023 7642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.125:36727 every 8 connection(s)
I20260925 00:48:46.674661 7575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
I20260925 00:48:46.677322 7643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:46.678218 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 7575
I20260925 00:48:46.681751 7643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.249.125@KRBTEST.COM
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.125@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:46.692860 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.683630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.0.249.126:46365)
0925 00:48:46.683789 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.683795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.684021 (+ 226us) server_negotiation.cc:407] Connection header received
0925 00:48:46.684151 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.684154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.684205 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.684286 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.684901 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.685559 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.686572 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.686796 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.686907 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.689531 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.689534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.689536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.689557 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.691153 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.691674 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.691678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.691679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.691763 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.692057 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.692060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.692062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.692267 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.692344 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.692671 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.692755 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:48:46.692898 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.679615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.0.249.125:36727)
0925 00:48:46.679997 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.680017 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.680061 (+ 44us) server_negotiation.cc:407] Connection header received
0925 00:48:46.680853 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.680875 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.681270 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.681636 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.682313 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.683689 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.684760 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.685322 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.685561 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.688362 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.688371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.688382 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.688408 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.689973 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.690463 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.690470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.690477 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.690617 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.690937 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.690939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.690940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.691330 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.691531 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.691776 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.692060 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
I20260925 00:48:46.692936 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.683553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42124)
0925 00:48:46.683894 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.683938 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.684059 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.684330 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.684334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.684654 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.684767 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.684773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.685681 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.685687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.686425 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.686437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.686935 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.686939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.687052 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.687555 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.687574 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.689347 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.691251 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.691258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.691263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.691547 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.691860 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.691862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.691864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.691949 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.692368 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.692373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.692487 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.692624 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.692774 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":117,"threads_started":1}
I20260925 00:48:46.694468 7643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.125@KRBTEST.COM for oryx/127.0.249.125@KRBTEST.COM
I20260925 00:48:46.701604 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.694903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.0.1:56940)
0925 00:48:46.694993 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.695004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.695088 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.695315 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.695317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.695475 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.695882 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.695888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.696568 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.696573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.697039 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.697045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.697499 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.697502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.697590 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.697919 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.697929 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.698960 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.700393 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.700396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.700397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.700578 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.700796 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.700798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.700799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.700865 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.701172 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.701174 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.701237 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.701307 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.701471 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 00:48:46.701627 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.694999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.0.249.125:36727)
0925 00:48:46.695085 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.695088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.695100 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:46.695167 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.695170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.695218 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.695306 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.695989 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.696466 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.697140 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.697368 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.697504 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.699083 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.699086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.699087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.699107 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.700305 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.700658 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.700660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.700662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.700725 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.700939 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.700941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.700942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.701102 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.701170 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.701357 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.701516 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 00:48:46.704071 7643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Bootstrap starting.
I20260925 00:48:46.706265 7643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:46.706946 7643 log.cc:824] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:46.708613 7643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: No bootstrap required, opened a new log
I20260925 00:48:46.711426 7643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } }
I20260925 00:48:46.711645 7643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:46.711730 7643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f8bacab3ba42be8f27d3ce89d5b46e, State: Initialized, Role: FOLLOWER
I20260925 00:48:46.712147 7643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } }
I20260925 00:48:46.712569 7649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } } }
I20260925 00:48:46.712669 7649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:48:46.713197 7643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: configured and running, proceeding with master startup.
W20260925 00:48:46.718132 7660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:48:46.718238 7660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:48:46.720466 7575 master_runner.cc:428] Detected that this master 46f8bacab3ba42be8f27d3ce89d5b46e is joining an existing cluster
I20260925 00:48:46.720515 7575 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.249.125:36727
I20260925 00:48:46.731184 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.722957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.0.249.125:36727)
0925 00:48:46.723049 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.723053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.723063 (+ 10us) server_negotiation.cc:407] Connection header received
0925 00:48:46.723092 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.723094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.723129 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.723203 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.723906 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.724657 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.725872 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.726217 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.726311 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.727381 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.727384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.727387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.727410 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.729309 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.729847 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.729851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.729853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.729946 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.730245 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.730248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.730250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.730445 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.730540 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.730849 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.731044 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":41}
I20260925 00:48:46.731199 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.722519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.0.1:56948)
0925 00:48:46.722838 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.722851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.722943 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.723240 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.723243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.723629 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.723779 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.723787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.724779 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.724783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.725756 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.725766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.726309 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.726312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.726397 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.726914 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.726925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.727229 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.729470 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.729475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.729479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.729729 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.730049 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.730056 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.730058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.730150 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.730565 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.730567 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.730649 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.730801 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.730988 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":4,"spinlock_wait_cycles":1280,"thread_start_us":97,"threads_started":1}
I20260925 00:48:46.732350 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.722481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42138)
0925 00:48:46.722799 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.722814 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.722916 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.723304 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.723306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.723609 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.723737 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.723743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.724850 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.724852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.725527 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.725535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.726029 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.726033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.726162 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.727137 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.727153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.727614 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.730573 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.730576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.730577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.730797 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.731130 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.731133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.731135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.731210 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.731855 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.731858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.731951 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.732047 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.732167 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":61,"spinlock_wait_cycles":896,"thread_start_us":95,"threads_started":1}
I20260925 00:48:46.732436 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.722547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.0.249.126:46365)
0925 00:48:46.722864 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.722871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.722885 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:46.723036 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.723050 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.723102 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.723156 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.723858 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.724687 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.725643 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.725910 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.726006 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.727734 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.727739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.727742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.727771 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.730154 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.730934 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.730937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.730939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.731020 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.731512 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.731516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.731518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.731720 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.731786 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.732141 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.732263 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.126@KRBTEST.COM for oryx/127.0.249.125@KRBTEST.COM
I20260925 00:48:46.745779 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.736790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.0.249.125:36727)
0925 00:48:46.736911 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.736915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.737245 (+ 330us) server_negotiation.cc:407] Connection header received
0925 00:48:46.737516 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.737519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.737562 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.737626 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.738391 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.738902 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.740118 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.740311 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.740387 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.742615 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.742619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.742622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.742646 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.744075 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.744484 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.744487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.744489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.744583 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.744829 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.744832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.744834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.745031 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.745120 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.745526 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.745629 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:48:46.745858 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.736765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.0.1:56950)
0925 00:48:46.737110 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.737165 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.737292 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.737683 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.737690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.738144 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.738271 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.738279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.738996 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.739000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.740012 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.740022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.740415 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.740420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.740536 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.740994 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.741008 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.742492 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.744175 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.744183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.744187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.744396 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.744676 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.744678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.744679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.744748 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.745141 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.745147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.745289 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.745480 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.745682 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20260925 00:48:46.747591 7111 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.249.125:36727: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.249.125"
port: 36727
}
attrs {
promote: true
}
}
dest_uuid: "c980451b649c44b78e24958553b2e521"
cas_config_opid_index: -1
I20260925 00:48:46.748257 7111 consensus_queue.cc:237] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } }
I20260925 00:48:46.750371 7670 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 46f8bacab3ba42be8f27d3ce89d5b46e (127.0.249.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } } }
I20260925 00:48:46.751315 7617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:46.752403 7670 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.249.125:36727
I20260925 00:48:46.752854 7575 master.cc:537] Master@127.0.249.125:36727 shutting down...
I20260925 00:48:46.752983 7617 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Refusing update from remote peer c980451b649c44b78e24958553b2e521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 00:48:46.753293 7670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } } }
I20260925 00:48:46.753540 7670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:46.753881 7672 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.0.249.124@KRBTEST.COM; defaulting to no policy
W20260925 00:48:46.755965 7096 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 00:48:46.756860 7096 consensus_peers.cc:487] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 -> Peer 46f8bacab3ba42be8f27d3ce89d5b46e (127.0.249.125:36727): Couldn't send request to peer 46f8bacab3ba42be8f27d3ce89d5b46e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.124@KRBTEST.COM" created.
I20260925 00:48:46.757421 7575 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:46.757725 7575 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:46.757846 7575 tablet_replica.cc:330] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.124.keytab.
Entry for principal oryx/127.0.249.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.124.keytab.
I20260925 00:48:46.772639 7575 master.cc:552] Master@127.0.249.125:36727 shutdown complete.
I20260925 00:48:46.772699 7575 master_runner.cc:305] Clearing existing system tablet
I20260925 00:48:46.772962 7575 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:46.772938 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.124:36861
--webserver_interface=127.0.249.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.124.keytab
--principal=oryx/127.0.249.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46365,127.0.249.125:36727,127.0.249.124:36861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:48:46.773036 7575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:46.778929 7575 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260925 00:48:46.779942 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.780184 7575 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:46.780315 7575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:46.780349 7575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal
uuid: "46f8bacab3ba42be8f27d3ce89d5b46e"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "f3aa7ab16049e0d9cd406cc726edae8f"
server_key_iv: "fc1bdcdd0190f56ae9ea412d87bc33f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.782492 7575 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:46.784750 7575 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 00:48:46.784839 7575 log.cc:1196] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 00:48:46.785149 7575 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Deleting consensus metadata
I20260925 00:48:46.785316 7575 master_runner.cc:315] Copying system tablet from 127.0.249.126:46365
I20260925 00:48:46.786497 7575 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.249.126:46365
I20260925 00:48:46.798265 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.787137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.0.249.126:46365)
0925 00:48:46.787266 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.787271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.789698 (+ 2427us) server_negotiation.cc:407] Connection header received
0925 00:48:46.789805 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.789809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.789864 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.789953 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.790553 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.791401 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.792709 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.793073 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.793174 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.794615 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.794619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.794622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.794650 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.796373 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.796926 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.796930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.796932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.797021 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.797319 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.797322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.797323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.797577 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.797674 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.797972 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.798109 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:46.798280 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.787058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42148)
0925 00:48:46.789577 (+ 2519us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.789592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.789719 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.789989 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.790065 (+ 76us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.790279 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.790414 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.790421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.791515 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.791519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.792564 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.792577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.793200 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.793205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.793324 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.793983 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.793999 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.794473 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.796538 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.796541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.796544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.796818 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.797123 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.797126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.797128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.797223 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.797678 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.797681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.797787 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.797906 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.798056 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2439,"thread_start_us":83,"threads_started":1}
I20260925 00:48:46.799337 7131 tablet_copy_service.cc:140] P c980451b649c44b78e24958553b2e521: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 46f8bacab3ba42be8f27d3ce89d5b46e ({username='oryx', principal='oryx/127.0.249.125@KRBTEST.COM'} at 127.0.0.1:42148)
I20260925 00:48:46.799542 7131 tablet_copy_service.cc:161] P c980451b649c44b78e24958553b2e521: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 46f8bacab3ba42be8f27d3ce89d5b46e at {username='oryx', principal='oryx/127.0.249.125@KRBTEST.COM'} at 127.0.0.1:42148: session id = 46f8bacab3ba42be8f27d3ce89d5b46e-00000000000000000000000000000000
I20260925 00:48:46.801203 7131 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 00:48:46.802896 7575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 00:48:46.805455 7575 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: tablet copy: Starting download of 0 data blocks...
I20260925 00:48:46.805653 7575 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: tablet copy: Starting download of 1 WAL segments...
I20260925 00:48:46.807401 7575 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 00:48:46.809240 7131 tablet_copy_service.cc:342] P c980451b649c44b78e24958553b2e521: Request end of tablet copy session 46f8bacab3ba42be8f27d3ce89d5b46e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.249.125@KRBTEST.COM'} at 127.0.0.1:42148
I20260925 00:48:46.809350 7131 tablet_copy_service.cc:433] P c980451b649c44b78e24958553b2e521: ending tablet copy session 46f8bacab3ba42be8f27d3ce89d5b46e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 46f8bacab3ba42be8f27d3ce89d5b46e
I20260925 00:48:46.816623 7575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:46.818357 7688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:46.818449 7685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:46.818786 7686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:46.819370 7575 server_base.cc:1034] running on GCE node
I20260925 00:48:46.819543 7575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:46.819833 7575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:46.820983 7575 hybrid_clock.cc:648] HybridClock initialized: now 1790297326820972 us; error 30 us; skew 500 ppm
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:46.822649 7575 init.cc:377] Logged in from keytab as oryx/127.0.249.125@KRBTEST.COM (short username oryx)
I20260925 00:48:46.823122 7575 webserver.cc:460] Webserver started at http://127.0.249.125:43155/ using document root <none> and password file <none>
I20260925 00:48:46.823302 7575 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:46.823350 7575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:46.824615 7575 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 00:48:46.825613 7695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.825867 7575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:46.825935 7575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal
uuid: "46f8bacab3ba42be8f27d3ce89d5b46e"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "f3aa7ab16049e0d9cd406cc726edae8f"
server_key_iv: "fc1bdcdd0190f56ae9ea412d87bc33f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.826011 7575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:46.858093 7575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:46.860376 7575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:46.865274 7575 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.125:36727
I20260925 00:48:46.865309 7747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.125:36727 every 8 connection(s)
I20260925 00:48:46.865916 7575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
I20260925 00:48:46.867722 7748 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:48:46.868793 7748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Bootstrap starting.
I20260925 00:48:46.878921 7748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:48:46.879215 7748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Bootstrap complete.
I20260925 00:48:46.879777 7748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } }
I20260925 00:48:46.879904 7748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46f8bacab3ba42be8f27d3ce89d5b46e, State: Initialized, Role: LEARNER
I20260925 00:48:46.880002 7748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } }
I20260925 00:48:46.880290 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } } }
I20260925 00:48:46.880410 7748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:46.880489 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: This master's current role is: LEARNER
I20260925 00:48:46.883383 7762 catalog_manager.cc:1259] Loaded cluster ID: f3da4ca6cc444f349e8b2510b228bff8
I20260925 00:48:46.883495 7762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: loading cluster ID for follower catalog manager: success
I20260925 00:48:46.885802 7762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: acquiring CA information for follower catalog manager: success
I20260925 00:48:46.887436 7762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 00:48:46.908821 7676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:46.909057 7676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:46.909097 7676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:46.913029 7676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:46.913077 7676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:46.913096 7676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:46.913112 7676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:46.913127 7676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:46.918524 7676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46365,127.0.249.125:36727,127.0.249.124:36861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.124.keytab
--principal=oryx/127.0.249.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.124:36861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.249.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:46.919497 7676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:46.920425 7676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:46.930594 7767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:46.930600 7768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:46.930567 7770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:46.931948 7676 server_base.cc:1034] running on GCE node
I20260925 00:48:46.932605 7676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:46.933794 7676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:46.934993 7676 hybrid_clock.cc:648] HybridClock initialized: now 1790297326934979 us; error 28 us; skew 500 ppm
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:46.937932 7676 init.cc:377] Logged in from keytab as oryx/127.0.249.124@KRBTEST.COM (short username oryx)
I20260925 00:48:46.938772 7676 webserver.cc:460] Webserver started at http://127.0.249.124:36375/ using document root <none> and password file <none>
I20260925 00:48:46.939440 7676 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:46.939502 7676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:46.939718 7676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:46.941764 7676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data/instance:
uuid: "56e211d840d247c8ae05c1b343bc2417"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "fb5296bed2fcd69f1d586016691b52d7"
server_key_iv: "9190dfee66a619ec3b4e1be2d1bec7f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.942265 7676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal/instance:
uuid: "56e211d840d247c8ae05c1b343bc2417"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "fb5296bed2fcd69f1d586016691b52d7"
server_key_iv: "9190dfee66a619ec3b4e1be2d1bec7f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.945815 7676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:46.948153 7777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:46.949497 7676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:46.949604 7676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal
uuid: "56e211d840d247c8ae05c1b343bc2417"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "fb5296bed2fcd69f1d586016691b52d7"
server_key_iv: "9190dfee66a619ec3b4e1be2d1bec7f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:46.949685 7676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:46.968339 7676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:46.970955 7676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:46.971098 7676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:46.977627 7676 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.124:36861
I20260925 00:48:46.977890 7829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.124:36861 every 8 connection(s)
I20260925 00:48:46.978624 7676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
I20260925 00:48:46.981472 7830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:46.982942 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 7676
I20260925 00:48:46.987243 7830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.249.124@KRBTEST.COM
Sep 25 00:48:46 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.124@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:46.998891 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.984288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.0.249.124:36861)
0925 00:48:46.984691 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.984706 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.984741 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:48:46.985493 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.985512 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.985894 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.986250 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.987118 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.988160 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.989117 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.989841 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.990001 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.992783 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.992791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.992803 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.992836 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.995545 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.996055 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.996062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.996067 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.996202 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.996527 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.996533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.996536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.997058 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.997257 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.997649 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.997925 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":94,"threads_started":1}
I20260925 00:48:46.999322 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.989060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.0.249.126:46365)
0925 00:48:46.989193 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:46.989197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:46.989371 (+ 174us) server_negotiation.cc:407] Connection header received
0925 00:48:46.989523 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:46.989526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:46.989574 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:46.989660 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:46.990335 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.990861 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.992035 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.992384 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.992504 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.995356 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:46.995360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:46.995363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:46.995393 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:46.997069 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.997987 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.997991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.997992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.998080 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:46.998367 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:46.998370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:46.998372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:46.998587 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:46.998681 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:46.999058 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 00:48:46.999183 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 00:48:46.999353 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:46.988861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42152)
0925 00:48:46.989241 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:46.989286 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:46.989427 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:46.989736 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:46.989741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:46.990062 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:46.990183 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.990189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.991026 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.991028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.991876 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:46.991890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:46.992580 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:46.992586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:46.992769 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:46.993444 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:46.993462 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:46.995211 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:46.997496 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.997509 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.997517 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.997893 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.998173 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:46.998175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:46.998177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:46.998264 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:46.998695 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:46.998700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:46.998799 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 00:48:46.998950 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 00:48:46.999177 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":163,"threads_started":1}
I20260925 00:48:47.000638 7830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.124@KRBTEST.COM for oryx/127.0.249.125@KRBTEST.COM
I20260925 00:48:47.008910 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.001624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.0.1:56958)
0925 00:48:47.001762 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.001774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.001835 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.002202 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.002204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.002324 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.002415 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.002420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.003241 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.003244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.003759 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.003765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.004244 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.004246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.004319 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.004810 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.004821 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.006069 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.007643 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.007645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.007647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.007844 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.008099 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.008102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.008103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.008185 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.008522 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.008525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.008624 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.008707 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.008801 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 00:48:47.008996 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.001791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.0.249.125:36727)
0925 00:48:47.002012 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.002015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.002032 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:48:47.002066 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.002068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.002105 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.002176 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.002511 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.003153 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.003852 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.004149 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.004226 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.006196 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.006198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.006200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.006224 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.007541 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.007927 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.007929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.007931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.007997 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.008265 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.008267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.008268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.008427 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.008500 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.008756 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.008849 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260925 00:48:47.009764 7830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.124@KRBTEST.COM for oryx/127.0.249.124@KRBTEST.COM
I20260925 00:48:47.017052 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.010213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.0.1:55124)
0925 00:48:47.010328 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.010342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.010411 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.010598 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.010600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.010733 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.010809 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.010813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.011886 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.011888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.012363 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.012370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.012781 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.012783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.012835 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.013217 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.013230 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.014430 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.015863 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.015865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.015867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.016049 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.016271 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.016273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.016316 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.016400 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.016711 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.016713 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.016778 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.016852 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.016937 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 00:48:47.017123 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.010269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.0.249.124:36861)
0925 00:48:47.010383 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.010386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.010398 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:47.010480 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.010482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.010513 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.010573 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.010945 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.011782 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.012462 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.012689 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.012768 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.014545 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.014548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.014550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.014576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.015772 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.016128 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.016130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.016132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.016196 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.016463 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.016465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.016466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.016632 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.016706 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.016907 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.016990 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:48:47.019923 7830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: Bootstrap starting.
I20260925 00:48:47.022163 7830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:47.022867 7830 log.cc:824] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:47.024777 7830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: No bootstrap required, opened a new log
I20260925 00:48:47.027927 7830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } }
I20260925 00:48:47.028146 7830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:47.028226 7830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e211d840d247c8ae05c1b343bc2417, State: Initialized, Role: FOLLOWER
I20260925 00:48:47.028682 7830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } }
I20260925 00:48:47.029124 7836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } } }
I20260925 00:48:47.029234 7836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:48:47.029855 7830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 00:48:47.034425 7847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:48:47.034503 7847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:48:47.037359 7676 master_runner.cc:428] Detected that this master 56e211d840d247c8ae05c1b343bc2417 is joining an existing cluster
I20260925 00:48:47.037456 7676 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.249.124:36861
I20260925 00:48:47.050659 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.039714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42158)
0925 00:48:47.040263 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.040278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.040417 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.040789 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.040791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.041086 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.041325 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.041333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.042399 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.042405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.043369 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.043378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.044112 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.044116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.044211 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.044880 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.044894 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.045236 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.047201 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.047204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.047206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.047776 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.048128 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.048131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.048134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.048221 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.048778 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.048782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.050158 (+ 1376us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.050288 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.050448 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":74,"thread_start_us":71,"threads_started":1}
I20260925 00:48:47.050662 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.039507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.0.249.126:46365)
0925 00:48:47.039621 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.039626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.040347 (+ 721us) server_negotiation.cc:407] Connection header received
0925 00:48:47.040545 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.040552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.040603 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.040652 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.041480 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.042259 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.043501 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.043893 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.043962 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.045375 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.045379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.045381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.045429 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.047086 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.047898 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.047904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.047910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.047994 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.048333 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.048336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.048338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.048560 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.048690 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.050364 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.050489 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:48:47.052683 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.039490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.0.1:56960)
0925 00:48:47.039973 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.039988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.040121 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.040482 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.040485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.040720 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.040865 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.040873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.045501 (+ 4628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.045506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.046501 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.046531 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.047129 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.047135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.047274 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.047754 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.047777 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.048186 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.050128 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.050131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.050132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.050359 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.050689 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.050691 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.050694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.051785 (+ 1091us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.052192 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.052193 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.052258 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.052362 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.052501 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"spinlock_wait_cycles":384,"thread_start_us":240,"threads_started":1}
I20260925 00:48:47.052706 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.039505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.0.249.125:36727)
0925 00:48:47.039624 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.039628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.040053 (+ 425us) server_negotiation.cc:407] Connection header received
0925 00:48:47.040235 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.040238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.040290 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.040341 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.041015 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.041894 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.046644 (+ 4750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.047004 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.047108 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.048312 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.048316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.048318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.048344 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.050021 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.050477 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.050481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.050482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.050574 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.051902 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.051905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.051907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.052080 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.052134 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.052410 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.052526 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 00:48:47.053293 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 7173
I20260925 00:48:47.061367 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:33341
--local_ip_for_outbound_sockets=127.0.249.65
--tserver_master_addrs=127.0.249.126:46365
--webserver_port=38379
--webserver_interface=127.0.249.65
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab
--principal=oryx/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.249.126:46365,127.0.249.125:36727
--tserver_master_addrs=127.0.249.126:46365,127.0.249.125:36727,127.0.249.124:36861 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:48:47.061453 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.040057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.0.1:55136)
0925 00:48:47.040374 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.040386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.040485 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.040789 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.040791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.041083 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.041327 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.041333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.042456 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.042458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.043584 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.043591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.044217 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.044220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.044325 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.057860 (+ 13535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.057874 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.058184 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.060046 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.060048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.060050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.060261 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.060496 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.060499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.060501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.060578 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.060945 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.060947 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.061016 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.061098 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.061217 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":88,"spinlock_wait_cycles":4864,"thread_start_us":88,"threads_started":1}
I20260925 00:48:47.062422 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.040080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.0.249.124:36861)
0925 00:48:47.040182 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.040188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.040439 (+ 251us) server_negotiation.cc:407] Connection header received
0925 00:48:47.040572 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.040574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.040608 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.040651 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.041466 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.042251 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.043709 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.044014 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.044094 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.058307 (+ 14213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.058312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.058315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.058338 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.059950 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.060354 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.060357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.060359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.060421 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.060651 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.060653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.060655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.060835 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.060916 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.062154 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.062288 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.126@KRBTEST.COM for oryx/127.0.249.124@KRBTEST.COM
I20260925 00:48:47.066658 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.056824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.0.1:55146)
0925 00:48:47.057003 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.057018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.057092 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.057364 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.057367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.057614 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.057754 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.057764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.059849 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.059853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.060735 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.060745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.061347 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.061351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.061463 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.062053 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.062067 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.063472 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.065177 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.065182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.065185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.065373 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.065691 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.065695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.065697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.065779 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.066135 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.066138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.066235 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.066331 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.066484 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 00:48:47.066673 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.056858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.0.249.124:36861)
0925 00:48:47.057122 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.057136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.057150 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:47.057216 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.057218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.057255 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.057349 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.058332 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.059709 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.060868 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.061226 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.062825 (+ 1599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.063608 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.063610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.063612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.063630 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.065062 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.065499 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.065503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.065504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.065598 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.065873 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.065876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.065877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.066041 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.066141 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.066392 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.066497 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260925 00:48:47.067445 7111 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.249.124:36861: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "56e211d840d247c8ae05c1b343bc2417"
member_type: NON_VOTER
last_known_addr {
host: "127.0.249.124"
port: 36861
}
attrs {
promote: true
}
}
dest_uuid: "c980451b649c44b78e24958553b2e521"
cas_config_opid_index: 5
I20260925 00:48:47.067988 7111 consensus_queue.cc:237] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } }
I20260925 00:48:47.069021 7670 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.069876 7804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:47.070546 7670 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.249.124:36861
I20260925 00:48:47.070717 7670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.070812 7670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:47.070987 7804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Refusing update from remote peer c980451b649c44b78e24958553b2e521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 00:48:47.071360 7676 master.cc:537] Master@127.0.249.124:36861 shutting down...
I20260925 00:48:47.072640 7671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 00:48:47.074805 7097 consensus_peers.cc:487] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 -> Peer 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124:36861): Couldn't send request to peer 56e211d840d247c8ae05c1b343bc2417. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 00:48:47.077941 7676 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:47.078217 7676 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:47.078320 7676 tablet_replica.cc:330] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: stopping tablet replica
I20260925 00:48:47.080225 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.071446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.0.249.125:36727)
0925 00:48:47.071571 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.071577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.071593 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:48:47.071642 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.071646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.071709 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.071808 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:47.076258 (+ 4450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.077013 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.078758 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.079391 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.079471 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.079624 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.079986 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.080090 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:47.080652 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.069140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.0.1:56966)
0925 00:48:47.069340 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.069353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.069441 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.073867 (+ 4426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.073873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.073896 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:47.073998 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.074005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.077279 (+ 3274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.077284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.078604 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.078615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.079721 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.079724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.079811 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.079861 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.080416 (+ 555us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.080491 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260925 00:48:47.081185 7722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 LEARNER]: Refusing update from remote peer c980451b649c44b78e24958553b2e521: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 00:48:47.081725 7671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 00:48:47.083748 7722 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.085667 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c980451b649c44b78e24958553b2e521. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.085783 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: This master's current role is: LEARNER
I20260925 00:48:47.085889 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: NON_VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: true } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.085965 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: This master's current role is: LEARNER
I20260925 00:48:47.086540 7676 master.cc:552] Master@127.0.249.124:36861 shutdown complete.
I20260925 00:48:47.086588 7676 master_runner.cc:305] Clearing existing system tablet
I20260925 00:48:47.086577 7670 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: attempting to promote NON_VOTER 46f8bacab3ba42be8f27d3ce89d5b46e to VOTER
I20260925 00:48:47.086824 7676 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:47.086876 7676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:47.087031 7670 consensus_queue.cc:237] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } }
I20260925 00:48:47.088183 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.087943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.0.1:55154)
0925 00:48:47.088043 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.088114 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 00:48:47.088979 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.088759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.0.1:55162)
0925 00:48:47.088868 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.088916 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 00:48:47.089126 7097 consensus_peers.cc:487] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 -> Peer 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124:36861): Couldn't send request to peer 56e211d840d247c8ae05c1b343bc2417. Status: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36861: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 00:48:47.089352 7722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 LEARNER]: Refusing update from remote peer c980451b649c44b78e24958553b2e521: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 00:48:47.089740 7671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 00:48:47.091161 7672 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 46f8bacab3ba42be8f27d3ce89d5b46e (127.0.249.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.092370 7672 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.092476 7672 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:47.092605 7672 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.092720 7672 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:47.093194 7722 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 46f8bacab3ba42be8f27d3ce89d5b46e (127.0.249.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.093705 7676 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260925 00:48:47.094825 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.094926 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:48:47.094977 7861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.095182 7676 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:47.095299 7676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:47.095336 7676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal
uuid: "56e211d840d247c8ae05c1b343bc2417"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "fb5296bed2fcd69f1d586016691b52d7"
server_key_iv: "9190dfee66a619ec3b4e1be2d1bec7f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:47.097076 7676 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:47.098891 7676 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 00:48:47.098994 7676 log.cc:1196] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 00:48:47.099243 7676 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: Deleting consensus metadata
I20260925 00:48:47.099427 7676 master_runner.cc:315] Copying system tablet from 127.0.249.126:46365
I20260925 00:48:47.100483 7676 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: tablet copy: Beginning tablet copy session from remote peer at address 127.0.249.126:46365
I20260925 00:48:47.108208 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.101014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.0.249.126:46365)
0925 00:48:47.101135 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.101140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.101332 (+ 192us) server_negotiation.cc:407] Connection header received
0925 00:48:47.101478 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.101481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.101526 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.101619 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.102053 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.102573 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.103382 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.103619 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.103727 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.104880 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.104883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.104886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.104910 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.106579 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.106997 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.107001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.107004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.107093 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.107367 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.107372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.107373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.107557 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.107665 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.107919 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.108041 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:48:47.108219 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.100974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42162)
0925 00:48:47.101248 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.101259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.101379 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.101636 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.101639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.101842 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.101936 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.101941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.102682 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.102687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.103265 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.103273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.103730 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.103733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.103828 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.104381 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.104402 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.104737 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.106682 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.106688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.106690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.106899 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.107190 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.107193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.107195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.107279 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.107653 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.107656 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.107759 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.107864 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.108003 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260925 00:48:47.109004 7131 tablet_copy_service.cc:140] P c980451b649c44b78e24958553b2e521: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 56e211d840d247c8ae05c1b343bc2417 ({username='oryx', principal='oryx/127.0.249.124@KRBTEST.COM'} at 127.0.0.1:42162)
I20260925 00:48:47.109119 7131 tablet_copy_service.cc:161] P c980451b649c44b78e24958553b2e521: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 56e211d840d247c8ae05c1b343bc2417 at {username='oryx', principal='oryx/127.0.249.124@KRBTEST.COM'} at 127.0.0.1:42162: session id = 56e211d840d247c8ae05c1b343bc2417-00000000000000000000000000000000
I20260925 00:48:47.110070 7131 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 00:48:47.111192 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 00:48:47.113071 7676 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: tablet copy: Starting download of 0 data blocks...
I20260925 00:48:47.113261 7676 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: tablet copy: Starting download of 1 WAL segments...
I20260925 00:48:47.114640 7676 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 00:48:47.116063 7131 tablet_copy_service.cc:342] P c980451b649c44b78e24958553b2e521: Request end of tablet copy session 56e211d840d247c8ae05c1b343bc2417-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.249.124@KRBTEST.COM'} at 127.0.0.1:42162
I20260925 00:48:47.116155 7131 tablet_copy_service.cc:433] P c980451b649c44b78e24958553b2e521: ending tablet copy session 56e211d840d247c8ae05c1b343bc2417-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 56e211d840d247c8ae05c1b343bc2417
I20260925 00:48:47.122802 7676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:47.124413 7868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:47.124480 7871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:47.125161 7676 server_base.cc:1034] running on GCE node
W20260925 00:48:47.125763 7869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:47.125973 7676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:47.126329 7676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:47.127496 7676 hybrid_clock.cc:648] HybridClock initialized: now 1790297327127480 us; error 31 us; skew 500 ppm
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:47.129089 7676 init.cc:377] Logged in from keytab as oryx/127.0.249.124@KRBTEST.COM (short username oryx)
I20260925 00:48:47.129617 7676 webserver.cc:460] Webserver started at http://127.0.249.124:45881/ using document root <none> and password file <none>
I20260925 00:48:47.129808 7676 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:47.129863 7676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:47.130949 7676 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:47.131881 7878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.132103 7676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:47.132174 7676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal
uuid: "56e211d840d247c8ae05c1b343bc2417"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "fb5296bed2fcd69f1d586016691b52d7"
server_key_iv: "9190dfee66a619ec3b4e1be2d1bec7f3"
server_key_version: "encryptionkey@0"
I20260925 00:48:47.132257 7676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:47.145550 7676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:47.147647 7676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:47.152489 7676 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.124:36861
I20260925 00:48:47.152506 7930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.124:36861 every 8 connection(s)
I20260925 00:48:47.153021 7676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
I20260925 00:48:47.154389 7931 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:48:47.155194 7931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: Bootstrap starting.
I20260925 00:48:47.163801 7931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:48:47.164103 7931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: Bootstrap complete.
I20260925 00:48:47.164585 7931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } }
I20260925 00:48:47.164716 7931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 56e211d840d247c8ae05c1b343bc2417, State: Initialized, Role: LEARNER
I20260925 00:48:47.164827 7931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } }
I20260925 00:48:47.165134 7934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.165289 7931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:47.165297 7934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: This master's current role is: LEARNER
I20260925 00:48:47.168912 7945 catalog_manager.cc:1259] Loaded cluster ID: f3da4ca6cc444f349e8b2510b228bff8
I20260925 00:48:47.168978 7945 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: loading cluster ID for follower catalog manager: success
I20260925 00:48:47.170445 7945 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: acquiring CA information for follower catalog manager: success
I20260925 00:48:47.171478 7945 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 00:48:47.226296 7857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:47.226752 7857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:47.226822 7857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:47.232925 7857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:47.233019 7857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:47.233140 7857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:47.241542 7857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab
--principal=oryx/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:33341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=38379
--tserver_master_addrs=127.0.249.126:46365,127.0.249.125:36727,127.0.249.124:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:47.242795 7857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:47.244009 7857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:47.254613 7953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:47.254642 7951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:47.254992 7950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:47.255157 7857 server_base.cc:1034] running on GCE node
I20260925 00:48:47.255512 7857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:47.256158 7857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:47.257340 7857 hybrid_clock.cc:648] HybridClock initialized: now 1790297327257314 us; error 44 us; skew 500 ppm
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:47.260179 7857 init.cc:377] Logged in from keytab as oryx/127.0.249.65@KRBTEST.COM (short username oryx)
I20260925 00:48:47.261021 7857 webserver.cc:460] Webserver started at http://127.0.249.65:38379/ using document root <none> and password file <none>
I20260925 00:48:47.261631 7857 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:47.261739 7857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:47.265327 7857 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:47.267264 7960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.268440 7857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:48:47.268570 7857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "1067f1241a5c4c69ba9acf81da15eed5"
format_stamp: "Formatted at 2026-09-25 00:48:45 on dist-test-slave-05m0"
server_key: "9a48f55cdae26d26838693a7ba819a08"
server_key_iv: "abe620ff210a95b79b8889e7dea4cb74"
server_key_version: "encryptionkey@0"
I20260925 00:48:47.269096 7857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:47.274619 7436 heartbeater.cc:503] Master 127.0.249.126:46365 was elected leader, sending a full tablet report...
I20260925 00:48:47.286724 7857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:47.289685 7857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:47.289888 7857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:47.290462 7857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:47.291291 7857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:47.291365 7857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.291440 7857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:47.291482 7857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.300148 7857 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:33341
I20260925 00:48:47.300203 8073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:33341 every 8 connection(s)
I20260925 00:48:47.301142 7857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:47.309213 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 7857
I20260925 00:48:47.309434 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 7307
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.125@KRBTEST.COM
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.124@KRBTEST.COM
I20260925 00:48:47.317020 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.302841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:52541 (local address 127.0.249.125:36727)
0925 00:48:47.302963 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.302967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.303863 (+ 896us) server_negotiation.cc:407] Connection header received
0925 00:48:47.304983 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.304987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.305050 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.305103 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.306481 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.307262 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.308623 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.308937 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.309033 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.312197 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.312203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.312210 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.312239 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.314024 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.314703 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.314707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.314709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.314790 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.315188 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.315191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.315194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.315440 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.315514 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.316740 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.316852 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:47.317775 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.304387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:35647 (local address 127.0.249.124:36861)
0925 00:48:47.304677 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.304686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.304698 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:47.305052 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.305056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.305105 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.305179 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.306481 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.307495 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.308394 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.308687 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.308776 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.313279 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.313282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.313285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.313316 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.315118 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.315562 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.315567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.315569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.315651 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.315949 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.315951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.315954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.316179 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.316281 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.317502 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.317622 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20260925 00:48:47.317934 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.302853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:53113 (local address 127.0.249.126:46365)
0925 00:48:47.302982 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.302986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.304187 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 00:48:47.304957 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.304961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.305023 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.305097 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.306470 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.307001 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.308660 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.308889 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.309030 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.312921 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.312927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.312933 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.312967 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.315552 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.316051 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.316057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.316062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.316149 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.316451 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.316454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.316456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.316703 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.316794 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.317554 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.317689 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:48:47.318107 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.303252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.249.65:52541)
0925 00:48:47.303695 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.303745 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.304752 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.305266 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.305269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.305719 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.306336 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.306345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.307368 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.307371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.308512 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.308526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.309146 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.309150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.309235 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.310042 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.310073 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.311985 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.314170 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.314179 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.314192 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.314587 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.314904 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.314907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.314909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.315076 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.315588 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.315594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.315876 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.317152 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.317270 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":197,"spinlock_wait_cycles":54528,"thread_start_us":126,"threads_started":1}
I20260925 00:48:47.318107 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.304214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.249.65:35647)
0925 00:48:47.304463 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.304473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.304752 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.305294 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.305296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.305727 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.306339 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.306345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.307597 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.307601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.308255 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.308262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.308937 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.308942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.309138 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.310042 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.310069 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.313113 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.315224 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.315230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.315235 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.315447 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.315747 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.315749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.315751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.315838 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.316329 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.316333 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.316451 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.316667 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.316989 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":97,"spinlock_wait_cycles":20096,"thread_start_us":75,"threads_started":1}
I20260925 00:48:47.319000 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.303252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.65:53113)
0925 00:48:47.304079 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.304094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.304751 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.305262 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.305268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.305717 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.306323 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.306337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.307143 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.307146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.308514 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.308523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.309176 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.309179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.309286 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.310274 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.310295 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.312453 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.315683 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.315689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.315694 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.315939 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.316250 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.316256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.316261 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.316351 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.316840 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.316846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.316946 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.318590 (+ 1644us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.318738 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":220,"thread_start_us":600,"threads_started":1}
I20260925 00:48:47.319360 8076 heartbeater.cc:347] Connected to a master server at 127.0.249.125:36727
I20260925 00:48:47.319389 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:45821
--local_ip_for_outbound_sockets=127.0.249.66
--tserver_master_addrs=127.0.249.126:46365
--webserver_port=34479
--webserver_interface=127.0.249.66
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab
--principal=oryx/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.249.126:46365,127.0.249.125:36727
--tserver_master_addrs=127.0.249.126:46365,127.0.249.125:36727,127.0.249.124:36861 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:48:47.319605 8076 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:47.319905 8075 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46365
I20260925 00:48:47.319988 8075 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:47.320194 8074 heartbeater.cc:347] Connected to a master server at 127.0.249.124:36861
I20260925 00:48:47.320168 8076 heartbeater.cc:511] Master 127.0.249.125:36727 requested a full tablet report, sending...
I20260925 00:48:47.320308 8074 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:47.320434 8074 heartbeater.cc:511] Master 127.0.249.124:36861 requested a full tablet report, sending...
I20260925 00:48:47.320164 8075 heartbeater.cc:511] Master 127.0.249.126:46365 requested a full tablet report, sending...
I20260925 00:48:47.321674 7111 ts_manager.cc:194] Re-registered known tserver with Master: 1067f1241a5c4c69ba9acf81da15eed5 (127.0.249.65:33341)
I20260925 00:48:47.321844 7895 ts_manager.cc:194] Registered new tserver with Master: 1067f1241a5c4c69ba9acf81da15eed5 (127.0.249.65:33341)
I20260925 00:48:47.322167 7712 ts_manager.cc:194] Registered new tserver with Master: 1067f1241a5c4c69ba9acf81da15eed5 (127.0.249.65:33341)
I20260925 00:48:47.322343 7111 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:53113
W20260925 00:48:47.440448 8082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:47.440749 8082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:47.440842 8082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:47.444934 8082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:47.445027 8082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:47.445117 8082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:47.450563 8082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab
--principal=oryx/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:45821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=34479
--tserver_master_addrs=127.0.249.126:46365,127.0.249.125:36727,127.0.249.124:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:47.451691 8082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:47.452730 8082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:47.462323 8090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:47.462327 8088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:47.462327 8087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:47.462677 8082 server_base.cc:1034] running on GCE node
I20260925 00:48:47.463080 8082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:47.463761 8082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:47.464912 8082 hybrid_clock.cc:648] HybridClock initialized: now 1790297327464898 us; error 27 us; skew 500 ppm
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:47.467864 8082 init.cc:377] Logged in from keytab as oryx/127.0.249.66@KRBTEST.COM (short username oryx)
I20260925 00:48:47.468750 8082 webserver.cc:460] Webserver started at http://127.0.249.66:34479/ using document root <none> and password file <none>
I20260925 00:48:47.469295 8082 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:47.469393 8082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:47.473208 8082 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:47.475600 8097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.476794 8082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:48:47.477211 7570 heartbeater.cc:503] Master 127.0.249.126:46365 was elected leader, sending a full tablet report...
I20260925 00:48:47.476997 8082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "8cd5d1b8797840909a3870baf96efb98"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "5162ca5deffc6191153b82e03d76ca5b"
server_key_iv: "eff351e4312bd2b48a4400d57161dfd3"
server_key_version: "encryptionkey@0"
I20260925 00:48:47.477681 8082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:47.494836 8082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:47.497725 8082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:47.497953 8082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:47.498558 8082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:47.499594 8082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:47.499646 8082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.499718 8082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:47.499754 8082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.509357 8082 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:45821
I20260925 00:48:47.509385 8210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:45821 every 8 connection(s)
I20260925 00:48:47.510444 8082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:47.511588 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 8082
I20260925 00:48:47.511766 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 7441
I20260925 00:48:47.520323 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:34397
--local_ip_for_outbound_sockets=127.0.249.67
--tserver_master_addrs=127.0.249.126:46365
--webserver_port=43557
--webserver_interface=127.0.249.67
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab
--principal=oryx/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.249.126:46365,127.0.249.125:36727
--tserver_master_addrs=127.0.249.126:46365,127.0.249.125:36727,127.0.249.124:36861 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:48:47.522853 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.517696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.0.1:55172)
0925 00:48:47.517942 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.517954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.518040 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.518544 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.518550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.518574 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:47.518694 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.518703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.519893 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.519899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.521100 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.521114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.522334 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.522339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.522435 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.522479 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.522606 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.522688 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260925 00:48:47.523313 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.517935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.0.249.124:36861)
0925 00:48:47.518248 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.518254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.518269 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:48:47.518315 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.518321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.518386 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.518797 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:47.518916 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.519761 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.521271 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.522204 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.522317 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.522502 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.523089 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.523172 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":88,"threads_started":1}
I20260925 00:48:47.523907 7905 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.125@KRBTEST.COM
I20260925 00:48:47.524336 7934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c980451b649c44b78e24958553b2e521. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.524519 7934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: This master's current role is: LEARNER
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.124@KRBTEST.COM
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:47.530968 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.514251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:35439 (local address 127.0.249.124:36861)
0925 00:48:47.514393 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.514397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.514992 (+ 595us) server_negotiation.cc:407] Connection header received
0925 00:48:47.515665 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.515671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.515730 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.515792 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.518055 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.518902 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.520964 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.521321 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.521423 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.525571 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.525575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.525579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.525609 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.528173 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.528943 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.528947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.528949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.529060 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.529566 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.529570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.529572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.529792 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.529873 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.530683 (+ 810us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.530809 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":64}
I20260925 00:48:47.530968 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.512819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:50677 (local address 127.0.249.125:36727)
0925 00:48:47.513416 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.513424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.514647 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 00:48:47.515700 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.515704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.515758 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.515817 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.518051 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.518886 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.521190 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.521568 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.521647 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.525533 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.525538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.525543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.525579 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.528104 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.528943 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.528947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.528949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.529060 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.529566 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.529570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.529571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.529801 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.529896 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.530683 (+ 787us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.530807 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473}
I20260925 00:48:47.531060 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.513528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:57047 (local address 127.0.249.126:46365)
0925 00:48:47.513668 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.513674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.514740 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 00:48:47.515649 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.515656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.515719 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.515777 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.519447 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.520222 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.523128 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.523484 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.523577 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.526474 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.526479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.526482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.526535 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.528478 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.529138 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.529144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.529148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.529241 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.529614 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.529621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.529625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.529872 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.529980 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.530696 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.530831 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:48:47.532907 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.512630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.249.66:50677)
0925 00:48:47.514538 (+ 1908us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.514550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.515407 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.516209 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.516218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.516928 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.517865 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.517887 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.519093 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.519096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.521061 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.521069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.521810 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.521814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.521911 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.522633 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.522660 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.525283 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.528250 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.528259 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.528278 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.528824 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.529188 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.529191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.529193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.529445 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.529965 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.529971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.530376 (+ 405us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.531725 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.531888 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1837,"mutex_wait_us":1507,"spinlock_wait_cycles":18048,"thread_start_us":143,"threads_started":1}
I20260925 00:48:47.532917 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.512629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.66:57047)
0925 00:48:47.514108 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.514162 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.515394 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.516209 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.516220 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.517158 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.519145 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.519154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.522153 (+ 2999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.522155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.522973 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.522987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.523744 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.523748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.523862 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.524713 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.524730 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.526321 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.528641 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.528647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.528651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.528992 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.529362 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.529367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.529371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.529494 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.530050 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.530054 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.530376 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.531269 (+ 893us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.531817 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"mutex_wait_us":1713,"spinlock_wait_cycles":12928,"thread_start_us":731,"threads_started":1}
I20260925 00:48:47.533183 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.514080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.249.66:35439)
0925 00:48:47.514892 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.514903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.515407 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.516209 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.516218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.516922 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.517865 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.517887 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.519093 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.519096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.520802 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.520828 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.521556 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.521560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.521769 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.523242 (+ 1473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.523258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.525287 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.528337 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.528341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.528344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.528818 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.529188 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.529191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.529193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.529445 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.529965 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.529971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.530376 (+ 405us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.531217 (+ 841us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.531709 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":441,"spinlock_wait_cycles":133376,"thread_start_us":88,"threads_started":1}
I20260925 00:48:47.534386 8213 heartbeater.cc:347] Connected to a master server at 127.0.249.125:36727
I20260925 00:48:47.534415 8211 heartbeater.cc:347] Connected to a master server at 127.0.249.124:36861
I20260925 00:48:47.534487 8212 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46365
I20260925 00:48:47.534698 8212 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:47.534698 8213 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:47.534698 8211 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:47.535295 8212 heartbeater.cc:511] Master 127.0.249.126:46365 requested a full tablet report, sending...
I20260925 00:48:47.535292 8213 heartbeater.cc:511] Master 127.0.249.125:36727 requested a full tablet report, sending...
I20260925 00:48:47.535295 8211 heartbeater.cc:511] Master 127.0.249.124:36861 requested a full tablet report, sending...
I20260925 00:48:47.536423 7111 ts_manager.cc:194] Re-registered known tserver with Master: 8cd5d1b8797840909a3870baf96efb98 (127.0.249.66:45821)
I20260925 00:48:47.536495 7712 ts_manager.cc:194] Registered new tserver with Master: 8cd5d1b8797840909a3870baf96efb98 (127.0.249.66:45821)
I20260925 00:48:47.536561 7895 ts_manager.cc:194] Registered new tserver with Master: 8cd5d1b8797840909a3870baf96efb98 (127.0.249.66:45821)
I20260925 00:48:47.537210 7111 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:57047
W20260925 00:48:47.637929 8219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:47.638155 8219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:47.638228 8219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:47.642241 8219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:47.642293 8219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:47.642391 8219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:47.647423 8219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab
--principal=oryx/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:34397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=43557
--tserver_master_addrs=127.0.249.126:46365,127.0.249.125:36727,127.0.249.124:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:47.648351 8219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:47.649446 8219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:47.658468 8224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:47.658552 8225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:47.658468 8227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:47.658957 8219 server_base.cc:1034] running on GCE node
I20260925 00:48:47.659339 8219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:47.659955 8219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:47.661145 8219 hybrid_clock.cc:648] HybridClock initialized: now 1790297327661131 us; error 25 us; skew 500 ppm
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:47.663817 8219 init.cc:377] Logged in from keytab as oryx/127.0.249.67@KRBTEST.COM (short username oryx)
I20260925 00:48:47.664541 8219 webserver.cc:460] Webserver started at http://127.0.249.67:43557/ using document root <none> and password file <none>
I20260925 00:48:47.665030 8219 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:47.665125 8219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:47.668967 8219 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:47.670848 8234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.671994 8219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:47.672113 8219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "0edc623c84004f16abb8d09a2cce0953"
format_stamp: "Formatted at 2026-09-25 00:48:46 on dist-test-slave-05m0"
server_key: "edaa1a04e404c5944a493ef0b2f8ba82"
server_key_iv: "d391fa02fc30f0559af080023ca80f64"
server_key_version: "encryptionkey@0"
I20260925 00:48:47.672674 8219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:47.694274 8219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:47.697517 8219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:47.697669 8219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:47.698211 8219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:47.698992 8219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:47.699046 8219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.699086 8219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:47.699106 8219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:47.707619 8219 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:34397
I20260925 00:48:47.707695 8347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:34397 every 8 connection(s)
I20260925 00:48:47.708658 8219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:47.710179 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 8219
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.125@KRBTEST.COM
I20260925 00:48:47.723322 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.712906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.0.249.126:46365)
0925 00:48:47.713042 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.713047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.713060 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:48:47.713110 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.713114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.713163 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.713240 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.714168 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.714989 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.716211 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.716540 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.716609 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.718873 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.718880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.718885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.718912 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.721266 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.721799 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.721802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.721804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.721883 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.722189 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.722194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.722198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.722557 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.722668 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.723001 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.723139 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:48:47.723663 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.712274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.0.249.125:36727)
0925 00:48:47.712847 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.712851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.712863 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:47.712914 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.712920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.712975 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.713060 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.714095 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.714877 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.717106 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.717509 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.717845 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.719756 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.719760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.719764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.719796 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.721682 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.722151 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.722157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.722162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.722245 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.722537 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.722541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.722543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.722744 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.722855 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.723384 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.723506 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":87,"threads_started":1}
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.124@KRBTEST.COM
Sep 25 00:48:47 dist-test-slave-05m0 krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:47.728615 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.713225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.0.249.124:36861)
0925 00:48:47.713343 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.713347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.713861 (+ 514us) server_negotiation.cc:407] Connection header received
0925 00:48:47.714111 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.714116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.714169 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.714219 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.715101 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.715898 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.718751 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.719105 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.719177 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.722427 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.722430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.722433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.722459 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.724684 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.725511 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.725520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.725525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.725631 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.726156 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.726161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.726163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.726979 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.727101 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.727821 (+ 720us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.728429 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:48:47.729992 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.710453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:55935 (local address 127.0.249.125:36727)
0925 00:48:47.710588 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.710593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.711676 (+ 1083us) server_negotiation.cc:407] Connection header received
0925 00:48:47.714113 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.714116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.714160 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.714205 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.716147 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.716887 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.718722 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.719062 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.719162 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.725950 (+ 6788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.725954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.725957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.725980 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.727581 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.728171 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.728175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.728177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.728251 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.728610 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.728613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.728614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.728809 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.728881 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.729756 (+ 875us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.729858 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:48:47.727027 7111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:47.731343 7111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:47.731748 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.710720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:35727 (local address 127.0.249.124:36861)
0925 00:48:47.710844 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.710850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.712706 (+ 1856us) server_negotiation.cc:407] Connection header received
0925 00:48:47.715918 (+ 3212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.715924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.715980 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.716047 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.719823 (+ 3776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.720647 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.722424 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.722787 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.722913 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.727205 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.727208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.727211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.727233 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.729360 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.730494 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.730498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.730500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.730574 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.730859 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.730862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.730864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.731087 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.731163 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.731497 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.731605 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:48:47.732220 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.712117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:39089)
0925 00:48:47.712865 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.712877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.715901 (+ 3024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.716234 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.716237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.716440 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.716548 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.716554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.717602 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.717605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.721488 (+ 3883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.721499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.723537 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.723544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.723658 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.725216 (+ 1558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.725235 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.727062 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.730303 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.730307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.730310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.730522 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.730810 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.730814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.730818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.730925 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.731357 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.731362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.731447 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.731529 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.731649 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":3010,"thread_start_us":70,"threads_started":1}
I20260925 00:48:47.732244 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.710945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.249.67:35727)
0925 00:48:47.712623 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.712635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.715808 (+ 3173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.718424 (+ 2616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.718430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.718631 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.719713 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.719723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.720752 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.720757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.722238 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.722247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.723049 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.723053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.723161 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.724621 (+ 1460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.724639 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.726743 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.730043 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.730047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.730052 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.730370 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.730676 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.730680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.730682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.730760 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.731218 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.731221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.731309 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.731877 (+ 568us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.732039 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1592,"mutex_wait_us":28,"spinlock_wait_cycles":46848,"thread_start_us":162,"threads_started":1}
I20260925 00:48:47.732738 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.713012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:39089 (local address 127.0.249.126:46365)
0925 00:48:47.713563 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.713568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.713583 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:48:47.715998 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.716003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.716056 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.716132 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.716675 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.717468 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.721592 (+ 4124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.721961 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.722067 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.727205 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.727208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.727211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.727233 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.730186 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.730624 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.730627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.730629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.730713 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.731026 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.731029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.731031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.731235 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.731317 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.732392 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.732529 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":91,"threads_started":1}
I20260925 00:48:47.733047 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.710945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.249.67:55935)
0925 00:48:47.711501 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.711569 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.713872 (+ 2303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.714582 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.714591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.715070 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:47.715726 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.715743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.717061 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.717064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.718605 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.718619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.719315 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.719319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.719518 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.720824 (+ 1306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:47.720843 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:47.724299 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:47.727702 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.727707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.727722 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.728061 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.728360 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:47.728363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:47.728364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:47.728511 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:47.729150 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:47.729158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:47.729473 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.730743 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.731092 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":49,"thread_start_us":669,"threads_started":1}
I20260925 00:48:47.733675 8349 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46365
I20260925 00:48:47.733727 8348 heartbeater.cc:347] Connected to a master server at 127.0.249.124:36861
I20260925 00:48:47.733911 8349 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:47.733917 8348 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:47.733922 8350 heartbeater.cc:347] Connected to a master server at 127.0.249.125:36727
I20260925 00:48:47.734058 8350 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:47.734517 8349 heartbeater.cc:511] Master 127.0.249.126:46365 requested a full tablet report, sending...
I20260925 00:48:47.734512 8350 heartbeater.cc:511] Master 127.0.249.125:36727 requested a full tablet report, sending...
I20260925 00:48:47.735181 7671 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: NON_VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: true } } }
I20260925 00:48:47.735325 7671 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:47.735586 7712 ts_manager.cc:194] Registered new tserver with Master: 0edc623c84004f16abb8d09a2cce0953 (127.0.249.67:34397)
I20260925 00:48:47.736045 7110 ts_manager.cc:194] Re-registered known tserver with Master: 0edc623c84004f16abb8d09a2cce0953 (127.0.249.67:34397)
I20260925 00:48:47.736691 7110 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:39089
I20260925 00:48:47.734513 8348 heartbeater.cc:511] Master 127.0.249.124:36861 requested a full tablet report, sending...
I20260925 00:48:47.737761 7895 ts_manager.cc:194] Registered new tserver with Master: 0edc623c84004f16abb8d09a2cce0953 (127.0.249.67:34397)
I20260925 00:48:47.739068 7671 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: attempting to promote NON_VOTER 56e211d840d247c8ae05c1b343bc2417 to VOTER
I20260925 00:48:47.739408 7671 consensus_queue.cc:237] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } }
I20260925 00:48:47.740728 7722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Refusing update from remote peer c980451b649c44b78e24958553b2e521: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 00:48:47.741077 7905 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 LEARNER]: Refusing update from remote peer c980451b649c44b78e24958553b2e521: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 00:48:47.741232 8365 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 00:48:47.741461 7671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 00:48:47.743008 8365 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:47.743539 7722 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:47.744235 7905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:47.744609 8365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:47.744829 8365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:47.745013 8365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:47.745180 8365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:47.745366 8365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:47.745553 8365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:47.745553 8375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:47.745719 8375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:48:47.746305 7934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c980451b649c44b78e24958553b2e521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:47.746425 7934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:48:47.753976 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.748468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:33341 (local address 127.0.0.1:48928)
0925 00:48:47.748591 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.748607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.748680 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.750128 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.750131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.750148 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:47.750245 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.750251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.751628 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.751631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.752579 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.752589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.753626 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.753630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.753718 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.753745 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.753790 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.753849 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":28}
I20260925 00:48:47.754626 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.749001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.0.249.65:33341)
0925 00:48:47.749610 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.749622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.749647 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:48:47.749727 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.749733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.749973 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.750163 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:47.750379 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.751511 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.752703 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.753492 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.754155 (+ 663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.754329 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.754371 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.754485 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":104,"threads_started":1}
I20260925 00:48:47.754880 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.749070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:45821 (local address 127.0.0.1:50708)
0925 00:48:47.750392 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.750406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.750487 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.750934 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.750936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.750953 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:47.751045 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.751052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.752150 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.752153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.753315 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.753324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.754498 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.754501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.754587 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.754609 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.754697 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.754771 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"thread_start_us":233,"threads_started":1}
I20260925 00:48:47.755635 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.748718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:34397 (local address 127.0.0.1:50784)
0925 00:48:47.749497 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.749511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.749607 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.751630 (+ 2023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.751632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.751647 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:47.751771 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.751776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.753073 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.753077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.754179 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.754189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.755364 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.755367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.755446 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.755463 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.755508 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.755550 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":18,"thread_start_us":563,"threads_started":1}
I20260925 00:48:47.755767 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.748998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.0.249.67:34397)
0925 00:48:47.750717 (+ 1719us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.750728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.750747 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:48:47.750819 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.750823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.750989 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.751138 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:47.751881 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.752960 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.754346 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.755252 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.755362 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.755491 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.755551 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.755638 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1626,"thread_start_us":95,"threads_started":1}
I20260925 00:48:47.757376 8008 tablet_service.cc:1467] Processing CreateTablet for tablet 9b948d23e1644f30803c506f77cb04fe (DEFAULT_TABLE table=test-table [id=eb114bb39d424de59df56f6c36805a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:47.758010 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.749214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.0.249.66:45821)
0925 00:48:47.749780 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.749788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.750509 (+ 721us) server_negotiation.cc:407] Connection header received
0925 00:48:47.750594 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.750599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.750781 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.750940 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:47.751134 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.752031 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.753469 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.754383 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.754484 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.754621 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.757768 (+ 3147us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.757869 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":116,"threads_started":1}
I20260925 00:48:47.758595 8008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b948d23e1644f30803c506f77cb04fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:47.759714 8282 tablet_service.cc:1467] Processing CreateTablet for tablet 9b948d23e1644f30803c506f77cb04fe (DEFAULT_TABLE table=test-table [id=eb114bb39d424de59df56f6c36805a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:47.760515 8282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b948d23e1644f30803c506f77cb04fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:47.760473 8145 tablet_service.cc:1467] Processing CreateTablet for tablet 9b948d23e1644f30803c506f77cb04fe (DEFAULT_TABLE table=test-table [id=eb114bb39d424de59df56f6c36805a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:47.761345 8145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b948d23e1644f30803c506f77cb04fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:47.768217 8381 tablet_bootstrap.cc:492] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: Bootstrap starting.
I20260925 00:48:47.769480 8383 tablet_bootstrap.cc:492] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: Bootstrap starting.
I20260925 00:48:47.770421 8381 tablet_bootstrap.cc:654] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:47.771167 8381 log.cc:824] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:47.771643 8383 tablet_bootstrap.cc:654] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:47.772459 8383 log.cc:824] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:47.773152 8381 tablet_bootstrap.cc:492] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: No bootstrap required, opened a new log
I20260925 00:48:47.773342 8381 ts_tablet_manager.cc:1397] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:48:47.774320 8383 tablet_bootstrap.cc:492] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: No bootstrap required, opened a new log
I20260925 00:48:47.774494 8383 ts_tablet_manager.cc:1397] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:48:47.776888 8381 raft_consensus.cc:348] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } }
I20260925 00:48:47.777119 8381 raft_consensus.cc:374] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:47.777174 8381 raft_consensus.cc:729] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1067f1241a5c4c69ba9acf81da15eed5, State: Initialized, Role: FOLLOWER
I20260925 00:48:47.777599 8381 consensus_queue.cc:260] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } }
I20260925 00:48:47.778617 8381 ts_tablet_manager.cc:1428] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:48:47.779403 8382 tablet_bootstrap.cc:492] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: Bootstrap starting.
I20260925 00:48:47.779853 8075 heartbeater.cc:503] Master 127.0.249.126:46365 was elected leader, sending a full tablet report...
I20260925 00:48:47.781683 8382 tablet_bootstrap.cc:654] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:47.782529 8382 log.cc:824] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:47.778790 8383 raft_consensus.cc:348] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } }
I20260925 00:48:47.783082 8383 raft_consensus.cc:374] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:47.783169 8383 raft_consensus.cc:729] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd5d1b8797840909a3870baf96efb98, State: Initialized, Role: FOLLOWER
I20260925 00:48:47.783622 8383 consensus_queue.cc:260] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } }
I20260925 00:48:47.784205 8212 heartbeater.cc:503] Master 127.0.249.126:46365 was elected leader, sending a full tablet report...
I20260925 00:48:47.784693 8382 tablet_bootstrap.cc:492] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: No bootstrap required, opened a new log
I20260925 00:48:47.784870 8382 ts_tablet_manager.cc:1397] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:48:47.785370 8383 ts_tablet_manager.cc:1428] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260925 00:48:47.790117 8382 raft_consensus.cc:348] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } }
I20260925 00:48:47.790292 8382 raft_consensus.cc:374] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:47.790345 8382 raft_consensus.cc:729] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0edc623c84004f16abb8d09a2cce0953, State: Initialized, Role: FOLLOWER
I20260925 00:48:47.790714 8382 consensus_queue.cc:260] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } }
I20260925 00:48:47.791323 8349 heartbeater.cc:503] Master 127.0.249.126:46365 was elected leader, sending a full tablet report...
I20260925 00:48:47.791566 8382 ts_tablet_manager.cc:1428] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260925 00:48:47.803037 8077 tablet.cc:2369] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:47.830008 8389 raft_consensus.cc:482] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:47.830222 8389 raft_consensus.cc:504] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } }
I20260925 00:48:47.831375 8389 leader_election.cc:288] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1067f1241a5c4c69ba9acf81da15eed5 (127.0.249.65:33341), 8cd5d1b8797840909a3870baf96efb98 (127.0.249.66:45821)
I20260925 00:48:47.835150 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.831626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:51771 (local address 127.0.249.65:33341)
0925 00:48:47.831823 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.831827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.831842 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:48:47.831914 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.831919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.831972 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.832041 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:47.832366 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.832917 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.834151 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.834558 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.834697 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.834750 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.834960 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.835028 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 00:48:47.835160 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.831545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:33341 (local address 127.0.249.67:51771)
0925 00:48:47.831661 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.831680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.831769 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.832078 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.832084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.832104 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:47.832259 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.832265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.833015 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.833019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.834035 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.834043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.834704 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.834708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.834814 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.834834 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.834891 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.834975 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":59}
I20260925 00:48:47.835198 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.831818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:60791 (local address 127.0.249.66:45821)
0925 00:48:47.831927 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.831931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.831941 (+ 10us) server_negotiation.cc:407] Connection header received
0925 00:48:47.831984 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.831986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.832046 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.832125 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:47.832471 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.833065 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.834184 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.834569 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.834722 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.834772 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.834991 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.835087 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 00:48:47.835180 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.831616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:45821 (local address 127.0.249.67:60791)
0925 00:48:47.831752 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:47.831767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:47.831844 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:47.832155 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:47.832158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:47.832176 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:47.832338 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.832346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.833162 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.833165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.834043 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:47.834052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.834723 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.834727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:47.834836 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.834852 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 00:48:47.834960 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 00:48:47.835027 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":34}
I20260925 00:48:47.835811 8028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b948d23e1644f30803c506f77cb04fe" candidate_uuid: "0edc623c84004f16abb8d09a2cce0953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1067f1241a5c4c69ba9acf81da15eed5" is_pre_election: true
I20260925 00:48:47.835813 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b948d23e1644f30803c506f77cb04fe" candidate_uuid: "0edc623c84004f16abb8d09a2cce0953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd5d1b8797840909a3870baf96efb98" is_pre_election: true
I20260925 00:48:47.836089 8028 raft_consensus.cc:2454] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0edc623c84004f16abb8d09a2cce0953 in term 0.
I20260925 00:48:47.836122 8165 raft_consensus.cc:2454] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0edc623c84004f16abb8d09a2cce0953 in term 0.
I20260925 00:48:47.836542 8237 leader_election.cc:302] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0edc623c84004f16abb8d09a2cce0953, 8cd5d1b8797840909a3870baf96efb98; no voters:
I20260925 00:48:47.836826 8389 raft_consensus.cc:2781] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:47.836912 8389 raft_consensus.cc:482] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:47.836954 8389 raft_consensus.cc:3037] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:47.838119 8389 raft_consensus.cc:504] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } }
I20260925 00:48:47.838485 8389 leader_election.cc:288] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [CANDIDATE]: Term 1 election: Requested vote from peers 1067f1241a5c4c69ba9acf81da15eed5 (127.0.249.65:33341), 8cd5d1b8797840909a3870baf96efb98 (127.0.249.66:45821)
I20260925 00:48:47.838788 8028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b948d23e1644f30803c506f77cb04fe" candidate_uuid: "0edc623c84004f16abb8d09a2cce0953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1067f1241a5c4c69ba9acf81da15eed5"
I20260925 00:48:47.838889 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b948d23e1644f30803c506f77cb04fe" candidate_uuid: "0edc623c84004f16abb8d09a2cce0953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd5d1b8797840909a3870baf96efb98"
I20260925 00:48:47.838913 8028 raft_consensus.cc:3037] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:47.839041 8165 raft_consensus.cc:3037] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:47.840188 8028 raft_consensus.cc:2454] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0edc623c84004f16abb8d09a2cce0953 in term 1.
I20260925 00:48:47.840287 8165 raft_consensus.cc:2454] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0edc623c84004f16abb8d09a2cce0953 in term 1.
I20260925 00:48:47.840543 8235 leader_election.cc:302] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0edc623c84004f16abb8d09a2cce0953, 1067f1241a5c4c69ba9acf81da15eed5; no voters:
I20260925 00:48:47.840706 8389 raft_consensus.cc:2781] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:47.840951 8389 raft_consensus.cc:686] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 1 LEADER]: Becoming Leader. State: Replica: 0edc623c84004f16abb8d09a2cce0953, State: Running, Role: LEADER
I20260925 00:48:47.841493 8389 consensus_queue.cc:237] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } }
I20260925 00:48:47.844797 7110 catalog_manager.cc:5495] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0edc623c84004f16abb8d09a2cce0953 (127.0.249.67). New cstate: current_term: 1 leader_uuid: "0edc623c84004f16abb8d09a2cce0953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } health_report { overall_health: UNKNOWN } } }
I20260925 00:48:47.876693 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.872675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.0.249.67:34397)
0925 00:48:47.872787 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.872791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.872816 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:48:47.873037 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.873040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.873096 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.873182 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:47.873458 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.874055 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.875236 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.875435 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.875503 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.875852 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:47.876297 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:47.876383 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.876480 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.876533 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 00:48:47.882673 8165 raft_consensus.cc:1261] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [term 1 FOLLOWER]: Refusing update from remote peer 0edc623c84004f16abb8d09a2cce0953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:47.882804 8028 raft_consensus.cc:1261] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [term 1 FOLLOWER]: Refusing update from remote peer 0edc623c84004f16abb8d09a2cce0953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:47.883350 8389 consensus_queue.cc:1035] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:47.883539 8390 consensus_queue.cc:1035] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:47.892786 8394 mvcc.cc:205] Tried to move back new op lower bound from 7333057854999543808 to 7333057854841405440. Current Snapshot: MvccSnapshot[applied={T|T < 7333057854999543808}]
I20260925 00:48:47.900799 7110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:47.900966 7110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:47.904943 7110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c980451b649c44b78e24958553b2e521: Sending DeleteTablet for 3 replicas of tablet 9b948d23e1644f30803c506f77cb04fe
I20260925 00:48:47.905841 8008 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b948d23e1644f30803c506f77cb04fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:47 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:48928
I20260925 00:48:47.905858 8145 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b948d23e1644f30803c506f77cb04fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:47 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:50708
I20260925 00:48:47.905889 8282 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b948d23e1644f30803c506f77cb04fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:47 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:50784
I20260925 00:48:47.906613 8405 tablet_replica.cc:330] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: stopping tablet replica
I20260925 00:48:47.906937 8405 raft_consensus.cc:2229] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:47.907289 8405 raft_consensus.cc:2258] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:47.908526 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 7079
I20260925 00:48:47.908694 8405 ts_tablet_manager.cc:1905] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:47.911770 8405 ts_tablet_manager.cc:1918] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:47.911875 8405 log.cc:1196] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/9b948d23e1644f30803c506f77cb04fe
I20260925 00:48:47.912215 8405 ts_tablet_manager.cc:1939] T 9b948d23e1644f30803c506f77cb04fe P 0edc623c84004f16abb8d09a2cce0953: Deleting consensus metadata
W20260925 00:48:47.916472 8237 connection.cc:535] server connection from 127.0.0.1:50784 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 00:48:47.918184 8404 tablet_replica.cc:330] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: stopping tablet replica
I20260925 00:48:47.918543 8404 raft_consensus.cc:2229] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:47.918921 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.907884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.0.249.125:36727)
0925 00:48:47.908060 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.908066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.908080 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:47.908115 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.908117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.908160 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.908247 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.909699 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.910584 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.912090 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.912512 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.912612 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.914960 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.914967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.914973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.915004 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.916806 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.917367 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.917371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.917373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.917479 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.917810 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.917816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.917820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.918055 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.918160 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.918647 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.918765 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 00:48:47.918903 8404 raft_consensus.cc:2258] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:47.919289 8403 tablet_replica.cc:330] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: stopping tablet replica
I20260925 00:48:47.919571 8403 raft_consensus.cc:2229] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:47.919874 8403 raft_consensus.cc:2258] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:47.924263 8404 ts_tablet_manager.cc:1905] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:47.927003 8403 ts_tablet_manager.cc:1905] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:47.930102 8403 ts_tablet_manager.cc:1918] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:47.930202 8403 log.cc:1196] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/9b948d23e1644f30803c506f77cb04fe
I20260925 00:48:47.932610 8404 ts_tablet_manager.cc:1918] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:47.932703 8404 log.cc:1196] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/9b948d23e1644f30803c506f77cb04fe
I20260925 00:48:47.932757 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.907944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.0.249.124:36861)
0925 00:48:47.908050 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.908054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.920870 (+ 12816us) server_negotiation.cc:407] Connection header received
0925 00:48:47.921053 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.921056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.921105 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.921155 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.922086 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.922873 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.923933 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.924241 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.924318 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.926428 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.926431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.926434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.926458 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.928981 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.931382 (+ 2401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.931385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.931387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.931480 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.931801 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.931804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.931806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.932001 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.932111 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.932458 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.932593 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:48:47.932873 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.920545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.0.249.124:36861)
0925 00:48:47.920666 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.920670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.921491 (+ 821us) server_negotiation.cc:407] Connection header received
0925 00:48:47.921597 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.921603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.921657 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.921743 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.922564 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.923281 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.925188 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.925589 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.925665 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.927639 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.927643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.927645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.927668 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.930841 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.931510 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.931514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.931516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.931598 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.931887 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.931890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.931892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.932088 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.932178 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.932574 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.932699 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:47.933153 8404 ts_tablet_manager.cc:1939] T 9b948d23e1644f30803c506f77cb04fe P 1067f1241a5c4c69ba9acf81da15eed5: Deleting consensus metadata
W20260925 00:48:47.934379 7962 connection.cc:441] server connection from 127.0.0.1:48928 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48928 (request call id 1) could send its response
I20260925 00:48:47.930567 8403 ts_tablet_manager.cc:1939] T 9b948d23e1644f30803c506f77cb04fe P 8cd5d1b8797840909a3870baf96efb98: Deleting consensus metadata
I20260925 00:48:47.935504 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:47.920697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.0.249.125:36727)
0925 00:48:47.920791 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:47.920795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:47.921766 (+ 971us) server_negotiation.cc:407] Connection header received
0925 00:48:47.921909 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:47.921912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:47.921951 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:47.921997 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:47.922901 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.923676 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.926156 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:47.926511 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:47.926582 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:47.929421 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:47.929426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:47.929429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:47.929453 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:47.931020 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.933952 (+ 2932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.933955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.933957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.934039 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:47.934569 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:47.934572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:47.934574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:47.934787 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:47.934896 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:47.935249 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 00:48:47.935364 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 00:48:47.935580 8101 connection.cc:441] server connection from 127.0.0.1:50708 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50708 (request call id 1) could send its response
I20260925 00:48:48.895766 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:48.895269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.65:36129)
0925 00:48:48.895540 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:48.895652 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
W20260925 00:48:48.896200 8075 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:46365 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:48.898571 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:48.898130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:53749)
0925 00:48:48.898392 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:48.898476 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":69,"threads_started":1}
W20260925 00:48:48.898953 8349 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:46365 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:48.901569 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:48.901088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.66:43483)
0925 00:48:48.901338 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:48.901467 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
W20260925 00:48:48.902031 8212 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:46365 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:49.405243 8417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c980451b649c44b78e24958553b2e521)
I20260925 00:48:49.405354 8418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c980451b649c44b78e24958553b2e521)
I20260925 00:48:49.405570 8417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } }
I20260925 00:48:49.405593 8418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } }
I20260925 00:48:49.406499 8418 leader_election.cc:288] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c980451b649c44b78e24958553b2e521 (127.0.249.126:46365), 46f8bacab3ba42be8f27d3ce89d5b46e (127.0.249.125:36727)
I20260925 00:48:49.406555 8417 leader_election.cc:288] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c980451b649c44b78e24958553b2e521 (127.0.249.126:46365), 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124:36861)
I20260925 00:48:49.407168 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.406655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42624)
0925 00:48:49.406971 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.407054 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":46,"threads_started":1}
W20260925 00:48:49.407347 7881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:49.407568 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.406692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42638)
0925 00:48:49.407404 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.407476 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":59,"threads_started":1}
W20260925 00:48:49.407883 7698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:49.408298 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.408054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42644)
0925 00:48:49.408172 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.408229 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 00:48:49.408414 7881 leader_election.cc:334] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c980451b649c44b78e24958553b2e521 (127.0.249.126:46365): Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:49.409034 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.408814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42658)
0925 00:48:49.408923 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.408973 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 00:48:49.409180 7698 leader_election.cc:334] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c980451b649c44b78e24958553b2e521 (127.0.249.126:46365): Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:49.411342 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.406832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:36727 (local address 127.0.0.1:56996)
0925 00:48:49.406971 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.406988 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:49.407096 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:49.407996 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:49.407999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:49.408015 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:49.408131 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:49.408140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:49.409177 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:49.409180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:49.410149 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:49.410159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:49.410831 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:49.410835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:49.410986 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:49.411038 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 00:48:49.411108 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:48:49.411184 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":19,"thread_start_us":50,"threads_started":1}
I20260925 00:48:49.411386 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.407167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.0.249.125:36727)
0925 00:48:49.407586 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:49.407590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:49.407603 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:48:49.407807 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:49.407810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:49.407849 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:49.407924 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:49.408263 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:49.409030 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:49.410266 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:49.410731 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:49.410808 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:49.410860 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:49.411128 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 00:48:49.411261 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":51,"threads_started":1}
I20260925 00:48:49.411525 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.407156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.0.249.124:36861)
0925 00:48:49.407366 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:49.407372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:49.407587 (+ 215us) server_negotiation.cc:407] Connection header received
0925 00:48:49.407809 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:49.407813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:49.407868 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:49.408175 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:49.408604 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:49.409306 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:49.410455 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:49.410979 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:49.411086 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:49.411142 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:49.411310 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 00:48:49.411379 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260925 00:48:49.411523 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.407002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36861 (local address 127.0.0.1:55208)
0925 00:48:49.407508 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.407522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:49.407696 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:49.408287 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:49.408290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:49.408313 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:49.408480 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:49.408507 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:49.409446 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:49.409449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:49.410320 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:49.410337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:49.411103 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:49.411107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:49.411187 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:49.411214 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:48:49.411288 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 00:48:49.411355 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":51,"threads_started":1}
I20260925 00:48:49.412189 7905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56e211d840d247c8ae05c1b343bc2417" is_pre_election: true
I20260925 00:48:49.412189 7722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e211d840d247c8ae05c1b343bc2417" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" is_pre_election: true
I20260925 00:48:49.412384 7722 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e211d840d247c8ae05c1b343bc2417 in term 1.
I20260925 00:48:49.412384 7905 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f8bacab3ba42be8f27d3ce89d5b46e in term 1.
I20260925 00:48:49.412731 7698 leader_election.cc:302] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46f8bacab3ba42be8f27d3ce89d5b46e, 56e211d840d247c8ae05c1b343bc2417; no voters: c980451b649c44b78e24958553b2e521
I20260925 00:48:49.412740 7880 leader_election.cc:302] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46f8bacab3ba42be8f27d3ce89d5b46e, 56e211d840d247c8ae05c1b343bc2417; no voters: c980451b649c44b78e24958553b2e521
I20260925 00:48:49.412942 8417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 00:48:49.413023 8417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c980451b649c44b78e24958553b2e521)
I20260925 00:48:49.413080 8417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:48:49.413084 8418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 00:48:49.413136 8418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c980451b649c44b78e24958553b2e521)
I20260925 00:48:49.413187 8418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:48:49.414254 8418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } }
I20260925 00:48:49.414253 8417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } }
I20260925 00:48:49.414628 8417 leader_election.cc:288] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 2 election: Requested vote from peers c980451b649c44b78e24958553b2e521 (127.0.249.126:46365), 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124:36861)
I20260925 00:48:49.414738 8418 leader_election.cc:288] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [CANDIDATE]: Term 2 election: Requested vote from peers c980451b649c44b78e24958553b2e521 (127.0.249.126:46365), 46f8bacab3ba42be8f27d3ce89d5b46e (127.0.249.125:36727)
I20260925 00:48:49.414923 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.414714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42672)
0925 00:48:49.414798 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.414864 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 00:48:49.415015 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.414734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42678)
0925 00:48:49.414908 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.414955 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 00:48:49.415076 7722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56e211d840d247c8ae05c1b343bc2417" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e"
I20260925 00:48:49.415155 7905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56e211d840d247c8ae05c1b343bc2417"
I20260925 00:48:49.415228 7722 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 56e211d840d247c8ae05c1b343bc2417 in current term 2: Already voted for candidate 46f8bacab3ba42be8f27d3ce89d5b46e in this term.
I20260925 00:48:49.415282 7905 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 46f8bacab3ba42be8f27d3ce89d5b46e in current term 2: Already voted for candidate 56e211d840d247c8ae05c1b343bc2417 in this term.
I20260925 00:48:49.415653 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.415459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42690)
0925 00:48:49.415550 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.415601 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 00:48:49.415767 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.415331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:42688)
0925 00:48:49.415646 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.415698 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266}
W20260925 00:48:49.415789 7881 leader_election.cc:334] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c980451b649c44b78e24958553b2e521 (127.0.249.126:46365): Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
W20260925 00:48:49.415861 7698 leader_election.cc:334] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c980451b649c44b78e24958553b2e521 (127.0.249.126:46365): Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:49.415859 7881 leader_election.cc:302] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e211d840d247c8ae05c1b343bc2417; no voters: 46f8bacab3ba42be8f27d3ce89d5b46e, c980451b649c44b78e24958553b2e521
I20260925 00:48:49.415928 7698 leader_election.cc:302] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46f8bacab3ba42be8f27d3ce89d5b46e; no voters: 56e211d840d247c8ae05c1b343bc2417, c980451b649c44b78e24958553b2e521
I20260925 00:48:49.415997 8418 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 00:48:49.416071 8417 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 00:48:49.897330 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.896843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.65:42715)
0925 00:48:49.897125 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.897219 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":120,"threads_started":1}
I20260925 00:48:49.899870 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.899493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:35599)
0925 00:48:49.899729 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.899787 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260925 00:48:49.902943 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:49.902558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.66:60253)
0925 00:48:49.902777 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:49.902843 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260925 00:48:50.898798 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:50.898238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.65:57085)
0925 00:48:50.898557 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:50.898657 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
I20260925 00:48:50.900811 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:50.900459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:51401)
0925 00:48:50.900687 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:50.900737 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260925 00:48:50.904017 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:50.903625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.66:34401)
0925 00:48:50.903814 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:50.903876 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260925 00:48:51.330296 8433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:51.330423 8433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } }
I20260925 00:48:51.330888 8433 leader_election.cc:288] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c980451b649c44b78e24958553b2e521 (127.0.249.126:46365), 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124:36861)
I20260925 00:48:51.331521 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.331126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:36910)
0925 00:48:51.331341 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.331421 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260925 00:48:51.331735 7905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56e211d840d247c8ae05c1b343bc2417" is_pre_election: true
I20260925 00:48:51.331902 7905 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f8bacab3ba42be8f27d3ce89d5b46e in term 2.
I20260925 00:48:51.332296 7698 leader_election.cc:302] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f8bacab3ba42be8f27d3ce89d5b46e, 56e211d840d247c8ae05c1b343bc2417; no voters:
I20260925 00:48:51.332362 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.332159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:36922)
0925 00:48:51.332262 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.332310 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 00:48:51.332466 7698 leader_election.cc:334] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c980451b649c44b78e24958553b2e521 (127.0.249.126:46365): Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:51.332468 8433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 00:48:51.332588 8433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:51.332638 8433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 2 FOLLOWER]: Advancing to term 3
I20260925 00:48:51.333773 8433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } }
I20260925 00:48:51.334095 8433 leader_election.cc:288] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 3 election: Requested vote from peers c980451b649c44b78e24958553b2e521 (127.0.249.126:46365), 56e211d840d247c8ae05c1b343bc2417 (127.0.249.124:36861)
I20260925 00:48:51.334400 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.334181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:36932)
0925 00:48:51.334260 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.334313 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 00:48:51.334609 7905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56e211d840d247c8ae05c1b343bc2417"
I20260925 00:48:51.334728 7905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 2 FOLLOWER]: Advancing to term 3
I20260925 00:48:51.335018 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.334834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:36936)
0925 00:48:51.334925 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.334972 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 00:48:51.335115 7698 leader_election.cc:334] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c980451b649c44b78e24958553b2e521 (127.0.249.126:46365): Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
I20260925 00:48:51.335738 7905 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f8bacab3ba42be8f27d3ce89d5b46e in term 3.
I20260925 00:48:51.336031 7698 leader_election.cc:302] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46f8bacab3ba42be8f27d3ce89d5b46e, 56e211d840d247c8ae05c1b343bc2417; no voters: c980451b649c44b78e24958553b2e521
I20260925 00:48:51.336216 8433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 3 FOLLOWER]: Leader election won for term 3
I20260925 00:48:51.336396 8433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [term 3 LEADER]: Becoming Leader. State: Replica: 46f8bacab3ba42be8f27d3ce89d5b46e, State: Running, Role: LEADER
I20260925 00:48:51.336784 8433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } }
I20260925 00:48:51.338490 8436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46f8bacab3ba42be8f27d3ce89d5b46e. Latest consensus state: current_term: 3 leader_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:51.338584 8436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:51.338887 8438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:51.340911 8438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:51.341238 8438 catalog_manager.cc:1259] Loaded cluster ID: f3da4ca6cc444f349e8b2510b228bff8
I20260925 00:48:51.341279 8438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:51.341851 8438 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:51.342202 8438 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Loaded TSK: 0
I20260925 00:48:51.342813 8438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:48:51.396565 7712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:51.399785 7712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:51.402752 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.402502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:36958)
0925 00:48:51.402620 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.402677 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 00:48:51.403153 7905 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [term 3 FOLLOWER]: Refusing update from remote peer 46f8bacab3ba42be8f27d3ce89d5b46e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 00:48:51.403589 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.403315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.0.1:36960)
0925 00:48:51.403457 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.403517 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 00:48:51.403658 8436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 00:48:51.403899 7698 consensus_peers.cc:487] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e -> Peer c980451b649c44b78e24958553b2e521 (127.0.249.126:46365): Couldn't send request to peer c980451b649c44b78e24958553b2e521. Status: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 00:48:51.405166 8440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46f8bacab3ba42be8f27d3ce89d5b46e. Latest consensus state: current_term: 3 leader_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:51.405301 8440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:48:51.406455 8433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:51.406589 8433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:51.406723 8440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c980451b649c44b78e24958553b2e521" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46365 } } peers { permanent_uuid: "46f8bacab3ba42be8f27d3ce89d5b46e" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 36727 } attrs { promote: false } } peers { permanent_uuid: "56e211d840d247c8ae05c1b343bc2417" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36861 } attrs { promote: false } } }
I20260925 00:48:51.406857 8440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e211d840d247c8ae05c1b343bc2417 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:48:51.416623 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.412114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:33341 (local address 127.0.0.1:55306)
0925 00:48:51.412509 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.412526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:51.412618 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:51.412935 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:51.412941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:51.412961 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:51.413088 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:51.413096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.414183 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.414188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:51.415338 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:51.415346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.416264 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.416267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:51.416355 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:51.416381 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:48:51.416423 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 00:48:51.416482 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260925 00:48:51.417218 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.412261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.0.249.65:33341)
0925 00:48:51.412664 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:51.412670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:51.412686 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:48:51.412741 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:51.412744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:51.412799 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:51.413207 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:51.413333 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.414065 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.415486 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.416134 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.416984 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:51.417036 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:51.417066 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 00:48:51.417117 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":76,"threads_started":1}
I20260925 00:48:51.417724 8008 tablet_service.cc:1467] Processing CreateTablet for tablet 30919c6174634f4d84efbd7e45eb5f5b (DEFAULT_TABLE table=test-table [id=a03fe2f624dd4adc9efa2b9f80e1b128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:51.417836 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.412725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:34397 (local address 127.0.0.1:55458)
0925 00:48:51.413565 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.413603 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:51.413689 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:51.414535 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:51.414538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:51.414554 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:51.414635 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:51.414640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.415436 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.415440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:51.416569 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:51.416576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.417453 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.417457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:51.417534 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:51.417565 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 00:48:51.417615 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 00:48:51.417683 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":94,"threads_started":1}
I20260925 00:48:51.417910 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.412862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.0.249.66:45821)
0925 00:48:51.413890 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:51.413894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:51.413905 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:48:51.413947 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:51.413949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:51.413987 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:51.414081 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:51.414329 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.415114 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.416497 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.417126 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.417228 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:51.417289 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:51.417689 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 00:48:51.417777 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":92,"threads_started":1}
I20260925 00:48:51.417837 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.412429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:45821 (local address 127.0.0.1:49734)
0925 00:48:51.413138 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.413151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:51.413232 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:51.414100 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:51.414103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:51.414119 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:51.414209 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:51.414215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.415224 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.415227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:51.416368 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:51.416377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.417353 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.417359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:51.417479 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:51.417502 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:48:51.417641 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 00:48:51.417689 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":67,"thread_start_us":87,"threads_started":1}
I20260925 00:48:51.418044 8008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30919c6174634f4d84efbd7e45eb5f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:51.418247 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.412862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.0.249.67:34397)
0925 00:48:51.414312 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:51.414317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:51.414330 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:48:51.414381 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:51.414383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:51.414420 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:51.414516 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:51.414750 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.415324 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.416679 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.417285 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.417917 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:51.417984 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:51.418023 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 00:48:51.418096 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1354,"thread_start_us":85,"threads_started":1}
I20260925 00:48:51.418557 8145 tablet_service.cc:1467] Processing CreateTablet for tablet 30919c6174634f4d84efbd7e45eb5f5b (DEFAULT_TABLE table=test-table [id=a03fe2f624dd4adc9efa2b9f80e1b128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:51.418815 8145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30919c6174634f4d84efbd7e45eb5f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:51.420269 8450 tablet_bootstrap.cc:492] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5: Bootstrap starting.
I20260925 00:48:51.421038 8282 tablet_service.cc:1467] Processing CreateTablet for tablet 30919c6174634f4d84efbd7e45eb5f5b (DEFAULT_TABLE table=test-table [id=a03fe2f624dd4adc9efa2b9f80e1b128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:51.421238 8450 tablet_bootstrap.cc:654] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:51.421283 8451 tablet_bootstrap.cc:492] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: Bootstrap starting.
I20260925 00:48:51.421312 8282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30919c6174634f4d84efbd7e45eb5f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:51.422127 8451 tablet_bootstrap.cc:654] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:51.422492 8450 tablet_bootstrap.cc:492] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5: No bootstrap required, opened a new log
I20260925 00:48:51.422591 8450 ts_tablet_manager.cc:1397] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:51.423310 8451 tablet_bootstrap.cc:492] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: No bootstrap required, opened a new log
I20260925 00:48:51.423395 8451 ts_tablet_manager.cc:1397] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:51.423457 8450 raft_consensus.cc:348] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } }
I20260925 00:48:51.423548 8450 raft_consensus.cc:374] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:51.423570 8450 raft_consensus.cc:729] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1067f1241a5c4c69ba9acf81da15eed5, State: Initialized, Role: FOLLOWER
I20260925 00:48:51.423696 8450 consensus_queue.cc:260] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } }
I20260925 00:48:51.423836 8454 tablet_bootstrap.cc:492] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: Bootstrap starting.
I20260925 00:48:51.423832 8451 raft_consensus.cc:348] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } }
I20260925 00:48:51.423956 8451 raft_consensus.cc:374] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:51.423995 8451 raft_consensus.cc:729] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd5d1b8797840909a3870baf96efb98, State: Initialized, Role: FOLLOWER
I20260925 00:48:51.423996 8450 ts_tablet_manager.cc:1428] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:51.424118 8451 consensus_queue.cc:260] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } }
I20260925 00:48:51.424441 8451 ts_tablet_manager.cc:1428] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:51.425459 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.424893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.66:55669)
0925 00:48:51.425137 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.425196 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260925 00:48:51.427489 8454 tablet_bootstrap.cc:654] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:51.428158 8213 heartbeater.cc:503] Master 127.0.249.125:36727 was elected leader, sending a full tablet report...
I20260925 00:48:51.428601 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.428338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.66:50575)
0925 00:48:51.428479 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.428535 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 00:48:51.428900 8454 tablet_bootstrap.cc:492] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: No bootstrap required, opened a new log
I20260925 00:48:51.428992 8454 ts_tablet_manager.cc:1397] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 00:48:51.429435 8454 raft_consensus.cc:348] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } }
I20260925 00:48:51.429551 8454 raft_consensus.cc:374] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:51.429593 8454 raft_consensus.cc:729] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0edc623c84004f16abb8d09a2cce0953, State: Initialized, Role: FOLLOWER
I20260925 00:48:51.429756 8076 heartbeater.cc:503] Master 127.0.249.125:36727 was elected leader, sending a full tablet report...
I20260925 00:48:51.429883 8454 consensus_queue.cc:260] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } }
I20260925 00:48:51.430223 8454 ts_tablet_manager.cc:1428] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:51.430619 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.424409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.65:56511)
0925 00:48:51.430454 (+ 6045us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.430515 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5983,"thread_start_us":107,"threads_started":1}
I20260925 00:48:51.431434 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.430608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:54213)
0925 00:48:51.431284 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.431352 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":82,"threads_started":1}
I20260925 00:48:51.431427 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.431118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.65:33001)
0925 00:48:51.431289 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.431356 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 00:48:51.432349 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.432065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:51815)
0925 00:48:51.432220 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.432266 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 00:48:51.432747 8350 heartbeater.cc:503] Master 127.0.249.125:36727 was elected leader, sending a full tablet report...
W20260925 00:48:51.462631 8351 tablet.cc:2369] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:51.489234 8460 raft_consensus.cc:482] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:51.489364 8460 raft_consensus.cc:504] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } }
I20260925 00:48:51.489797 8460 leader_election.cc:288] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1067f1241a5c4c69ba9acf81da15eed5 (127.0.249.65:33341), 8cd5d1b8797840909a3870baf96efb98 (127.0.249.66:45821)
I20260925 00:48:51.490237 8028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30919c6174634f4d84efbd7e45eb5f5b" candidate_uuid: "0edc623c84004f16abb8d09a2cce0953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1067f1241a5c4c69ba9acf81da15eed5" is_pre_election: true
I20260925 00:48:51.490236 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30919c6174634f4d84efbd7e45eb5f5b" candidate_uuid: "0edc623c84004f16abb8d09a2cce0953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd5d1b8797840909a3870baf96efb98" is_pre_election: true
I20260925 00:48:51.490391 8165 raft_consensus.cc:2454] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0edc623c84004f16abb8d09a2cce0953 in term 0.
I20260925 00:48:51.490391 8028 raft_consensus.cc:2454] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0edc623c84004f16abb8d09a2cce0953 in term 0.
I20260925 00:48:51.490748 8237 leader_election.cc:302] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0edc623c84004f16abb8d09a2cce0953, 8cd5d1b8797840909a3870baf96efb98; no voters:
I20260925 00:48:51.490970 8460 raft_consensus.cc:2781] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:51.491041 8460 raft_consensus.cc:482] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:51.491103 8460 raft_consensus.cc:3037] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:51.492117 8460 raft_consensus.cc:504] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } }
I20260925 00:48:51.492445 8460 leader_election.cc:288] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [CANDIDATE]: Term 1 election: Requested vote from peers 1067f1241a5c4c69ba9acf81da15eed5 (127.0.249.65:33341), 8cd5d1b8797840909a3870baf96efb98 (127.0.249.66:45821)
I20260925 00:48:51.492800 8028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30919c6174634f4d84efbd7e45eb5f5b" candidate_uuid: "0edc623c84004f16abb8d09a2cce0953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1067f1241a5c4c69ba9acf81da15eed5"
I20260925 00:48:51.492787 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30919c6174634f4d84efbd7e45eb5f5b" candidate_uuid: "0edc623c84004f16abb8d09a2cce0953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd5d1b8797840909a3870baf96efb98"
I20260925 00:48:51.492923 8165 raft_consensus.cc:3037] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:51.492934 8028 raft_consensus.cc:3037] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:51.493912 8165 raft_consensus.cc:2454] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0edc623c84004f16abb8d09a2cce0953 in term 1.
I20260925 00:48:51.494012 8028 raft_consensus.cc:2454] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0edc623c84004f16abb8d09a2cce0953 in term 1.
I20260925 00:48:51.494261 8237 leader_election.cc:302] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0edc623c84004f16abb8d09a2cce0953, 8cd5d1b8797840909a3870baf96efb98; no voters:
I20260925 00:48:51.494439 8460 raft_consensus.cc:2781] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:51.494640 8460 raft_consensus.cc:686] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 1 LEADER]: Becoming Leader. State: Replica: 0edc623c84004f16abb8d09a2cce0953, State: Running, Role: LEADER
I20260925 00:48:51.495021 8460 consensus_queue.cc:237] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } }
I20260925 00:48:51.495419 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.495108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:58797)
0925 00:48:51.495282 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.495341 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 00:48:51.496712 7712 catalog_manager.cc:5495] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0edc623c84004f16abb8d09a2cce0953 (127.0.249.67). New cstate: current_term: 1 leader_uuid: "0edc623c84004f16abb8d09a2cce0953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0edc623c84004f16abb8d09a2cce0953" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34397 } health_report { overall_health: HEALTHY } } }
I20260925 00:48:51.514708 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.511141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.0.249.67:34397)
0925 00:48:51.511287 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:51.511292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:51.511308 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:48:51.511362 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:51.511365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:51.511424 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:51.511507 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:51.511765 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.512670 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.513555 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:51.513948 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:51.514062 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:51.514263 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:51.514352 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:51.514422 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:51.514502 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 00:48:51.514562 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 00:48:51.516810 8214 tablet.cc:2369] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:51.516835 8165 raft_consensus.cc:1261] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [term 1 FOLLOWER]: Refusing update from remote peer 0edc623c84004f16abb8d09a2cce0953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:51.516888 8028 raft_consensus.cc:1261] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5 [term 1 FOLLOWER]: Refusing update from remote peer 0edc623c84004f16abb8d09a2cce0953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:51.517439 8460 consensus_queue.cc:1035] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1067f1241a5c4c69ba9acf81da15eed5" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 33341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:51.517761 8460 consensus_queue.cc:1035] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd5d1b8797840909a3870baf96efb98" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 45821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:51.519321 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.518996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.65:60215)
0925 00:48:51.519166 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.519234 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 00:48:51.520162 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.519894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.66:46259)
0925 00:48:51.520010 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.520074 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 00:48:51.522537 8468 mvcc.cc:205] Tried to move back new op lower bound from 7333057869888897024 to 7333057869804994560. Current Snapshot: MvccSnapshot[applied={T|T < 7333057869888897024}]
I20260925 00:48:51.522856 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.522347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.66:52847)
0925 00:48:51.522729 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.522782 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318}
I20260925 00:48:51.523900 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.523539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:41433)
0925 00:48:51.523760 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.523818 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260925 00:48:51.525295 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.524492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.65:42117)
0925 00:48:51.525155 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.525225 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589}
I20260925 00:48:51.525767 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.524832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46365 (local address 127.0.249.67:45745)
0925 00:48:51.525652 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.525701 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":752}
I20260925 00:48:51.529958 7712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:51.530125 7712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:51.533638 7712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46f8bacab3ba42be8f27d3ce89d5b46e: Sending DeleteTablet for 3 replicas of tablet 30919c6174634f4d84efbd7e45eb5f5b
I20260925 00:48:51.534375 8145 tablet_service.cc:1514] Processing DeleteTablet for tablet 30919c6174634f4d84efbd7e45eb5f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:51 UTC) from {username='oryx', principal='oryx/127.0.249.125@KRBTEST.COM'} at 127.0.0.1:49734
I20260925 00:48:51.534381 8008 tablet_service.cc:1514] Processing DeleteTablet for tablet 30919c6174634f4d84efbd7e45eb5f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:51 UTC) from {username='oryx', principal='oryx/127.0.249.125@KRBTEST.COM'} at 127.0.0.1:55306
I20260925 00:48:51.534509 8282 tablet_service.cc:1514] Processing DeleteTablet for tablet 30919c6174634f4d84efbd7e45eb5f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:51 UTC) from {username='oryx', principal='oryx/127.0.249.125@KRBTEST.COM'} at 127.0.0.1:55458
I20260925 00:48:51.534838 8475 tablet_replica.cc:330] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: stopping tablet replica
I20260925 00:48:51.535017 8476 tablet_replica.cc:330] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5: stopping tablet replica
I20260925 00:48:51.535032 8475 raft_consensus.cc:2229] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:51.535080 8477 tablet_replica.cc:330] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: stopping tablet replica
I20260925 00:48:51.535173 8475 raft_consensus.cc:2258] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:51.535235 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 7857
I20260925 00:48:51.535233 8476 raft_consensus.cc:2229] T 30919c6174634f4d84efbd7e45eb5f5b P 1067f1241a5c4c69ba9acf81da15eed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:51.535269 8477 raft_consensus.cc:2229] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:51.536907 8475 ts_tablet_manager.cc:1905] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:51.539248 8475 ts_tablet_manager.cc:1918] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:51.539356 8475 log.cc:1196] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/30919c6174634f4d84efbd7e45eb5f5b
I20260925 00:48:51.539625 8477 raft_consensus.cc:2258] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:51.539716 8475 ts_tablet_manager.cc:1939] T 30919c6174634f4d84efbd7e45eb5f5b P 8cd5d1b8797840909a3870baf96efb98: Deleting consensus metadata
I20260925 00:48:51.540432 8477 ts_tablet_manager.cc:1905] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:51.540716 7698 catalog_manager.cc:4841] TS 8cd5d1b8797840909a3870baf96efb98 (127.0.249.66:45821): tablet 30919c6174634f4d84efbd7e45eb5f5b (table test-table [id=a03fe2f624dd4adc9efa2b9f80e1b128]) successfully deleted
I20260925 00:48:51.541142 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 8082
I20260925 00:48:51.542562 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:51.542218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:33341 (local address 127.0.0.1:55322)
0925 00:48:51.542385 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:51.542454 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 00:48:51.542783 7696 catalog_manager.cc:4584] TS 1067f1241a5c4c69ba9acf81da15eed5 (127.0.249.65:33341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30919c6174634f4d84efbd7e45eb5f5b: Network error: Client connection negotiation failed: client connection to 127.0.249.65:33341: connect: Connection refused (error 111)
I20260925 00:48:51.543437 8477 ts_tablet_manager.cc:1918] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:51.543543 8477 log.cc:1196] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/30919c6174634f4d84efbd7e45eb5f5b
I20260925 00:48:51.543846 8477 ts_tablet_manager.cc:1939] T 30919c6174634f4d84efbd7e45eb5f5b P 0edc623c84004f16abb8d09a2cce0953: Deleting consensus metadata
I20260925 00:48:51.544548 7696 catalog_manager.cc:4841] TS 0edc623c84004f16abb8d09a2cce0953 (127.0.249.67:34397): tablet 30919c6174634f4d84efbd7e45eb5f5b (table test-table [id=a03fe2f624dd4adc9efa2b9f80e1b128]) successfully deleted
I20260925 00:48:51.548261 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 8219
I20260925 00:48:51.554724 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 7575
I20260925 00:48:51.560293 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 7676
2026-09-25T00:48:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7140 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:51 dist-test-slave-05m0 krb5kdc[8480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:51 dist-test-slave-05m0 krb5kdc[8480](info): set up 2 sockets
Sep 25 00:48:51 dist-test-slave-05m0 krb5kdc[8480](info): commencing operation
krb5kdc: starting...
W20260925 00:48:52.606206 997 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:52 dist-test-slave-05m0 krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.126.keytab.
Entry for principal oryx/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.126.keytab.
I20260925 00:48:52.724164 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:44529
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.126.keytab
--principal=oryx/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:44529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:52.834846 8493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:52.835121 8493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:52.835162 8493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:52.839097 8493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:52.839157 8493 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 00:48:52.839179 8493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:52.839200 8493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:52.839218 8493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:52.839241 8493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:52.844331 8493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:44529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.126.keytab
--principal=oryx/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:44529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:52.845309 8493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:52.846328 8493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:52.854982 8501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:52.855093 8498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:52.855204 8493 server_base.cc:1034] running on GCE node
W20260925 00:48:52.854981 8499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:52.855757 8493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:52.856817 8493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:52.857980 8493 hybrid_clock.cc:648] HybridClock initialized: now 1790297332857966 us; error 29 us; skew 500 ppm
Sep 25 00:48:52 dist-test-slave-05m0 krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297332, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:52.860695 8493 init.cc:377] Logged in from keytab as oryx/127.0.249.126@KRBTEST.COM (short username oryx)
I20260925 00:48:52.861557 8493 webserver.cc:460] Webserver started at http://127.0.249.126:40229/ using document root <none> and password file <none>
I20260925 00:48:52.862107 8493 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:52.862166 8493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:52.862320 8493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:52.864194 8493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "804ccc135e574f848611ef0b4871da13"
format_stamp: "Formatted at 2026-09-25 00:48:52 on dist-test-slave-05m0"
server_key: "036e3f9e454ade195b39994934ebe370"
server_key_iv: "f7e3e4d806e5b79d786eed9a4ed2c84b"
server_key_version: "encryptionkey@0"
I20260925 00:48:52.864696 8493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "804ccc135e574f848611ef0b4871da13"
format_stamp: "Formatted at 2026-09-25 00:48:52 on dist-test-slave-05m0"
server_key: "036e3f9e454ade195b39994934ebe370"
server_key_iv: "f7e3e4d806e5b79d786eed9a4ed2c84b"
server_key_version: "encryptionkey@0"
I20260925 00:48:52.868153 8493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260925 00:48:52.870360 8508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:52.871521 8493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 00:48:52.871677 8493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "804ccc135e574f848611ef0b4871da13"
format_stamp: "Formatted at 2026-09-25 00:48:52 on dist-test-slave-05m0"
server_key: "036e3f9e454ade195b39994934ebe370"
server_key_iv: "f7e3e4d806e5b79d786eed9a4ed2c84b"
server_key_version: "encryptionkey@0"
I20260925 00:48:52.871775 8493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:52.890552 8493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:52.893621 8493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:52.893812 8493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:52.901458 8493 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:44529
I20260925 00:48:52.901486 8570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:44529 every 8 connection(s)
I20260925 00:48:52.902356 8493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:52.904903 8571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:52.910125 8571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: Bootstrap starting.
I20260925 00:48:52.910190 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 8493
I20260925 00:48:52.910327 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:52.910601 997 external_mini_cluster.cc:1428] Setting key 294415b46f60f4337113b3631ec1c95a
I20260925 00:48:52.912695 8571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:52.913597 8571 log.cc:824] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:52.915633 8571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: No bootstrap required, opened a new log
Sep 25 00:48:52 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:52.919765 8571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804ccc135e574f848611ef0b4871da13" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44529 } }
I20260925 00:48:52.920044 8571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:52.920131 8571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804ccc135e574f848611ef0b4871da13, State: Initialized, Role: FOLLOWER
I20260925 00:48:52.920629 8571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804ccc135e574f848611ef0b4871da13" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44529 } }
I20260925 00:48:52.920809 8571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:52.920885 8571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:52.921011 8571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:52.921919 8571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804ccc135e574f848611ef0b4871da13" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44529 } }
I20260925 00:48:52.922250 8571 leader_election.cc:302] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 804ccc135e574f848611ef0b4871da13; no voters:
I20260925 00:48:52.922573 8571 leader_election.cc:288] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:52.922636 8576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:52.922824 8576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [term 1 LEADER]: Becoming Leader. State: Replica: 804ccc135e574f848611ef0b4871da13, State: Running, Role: LEADER
I20260925 00:48:52.923156 8576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804ccc135e574f848611ef0b4871da13" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44529 } }
I20260925 00:48:52.923831 8571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:52.925684 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:52.911858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.0.249.126:44529)
0925 00:48:52.912266 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:52.912285 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:52.912325 (+ 40us) server_negotiation.cc:407] Connection header received
0925 00:48:52.913214 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:52.913236 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:52.913698 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:52.914060 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:52.914825 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:52.916223 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:52.917086 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:52.917837 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:52.918000 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:52.920589 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:52.920595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:52.920610 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:52.920642 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:52.922662 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:52.923138 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:52.923144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:52.923149 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:52.923302 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:52.923615 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:52.923619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:52.923621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:52.924072 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:52.924252 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:52.924542 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 00:48:52.924868 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":124,"threads_started":1}
I20260925 00:48:52.926122 8577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "804ccc135e574f848611ef0b4871da13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804ccc135e574f848611ef0b4871da13" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44529 } } }
I20260925 00:48:52.927661 8577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:52.926136 8578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 804ccc135e574f848611ef0b4871da13. Latest consensus state: current_term: 1 leader_uuid: "804ccc135e574f848611ef0b4871da13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804ccc135e574f848611ef0b4871da13" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44529 } } }
I20260925 00:48:52.928300 8578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13 [sys.catalog]: This master's current role is: LEADER
W20260925 00:48:52.929055 8590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:48:52.929114 8590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:48:52.929191 8593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:52.930105 8593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:52.935382 8593 catalog_manager.cc:1347] Generated new cluster ID: 700555ae06974696a55e362b71572265
I20260925 00:48:52.935444 8593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:52.945307 8593 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:52.946574 8593 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:52.954737 8593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: Generated new TSK 0
I20260925 00:48:52.955567 8593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab.
Entry for principal oryx/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab.
I20260925 00:48:52.986478 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:44529
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab
--principal=oryx/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:53.143888 8597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:53.144142 8597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:53.144191 8597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:53.148438 8597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:53.148492 8597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:53.148528 8597 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:48:53.148561 8597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:53.153492 8597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab
--principal=oryx/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.249.126:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:53.154376 8597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:53.155344 8597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:53.165069 8605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:53.164979 8603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:53.164979 8602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:53.165297 8597 server_base.cc:1034] running on GCE node
I20260925 00:48:53.165711 8597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:53.166332 8597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:53.167512 8597 hybrid_clock.cc:648] HybridClock initialized: now 1790297333167494 us; error 29 us; skew 500 ppm
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297333, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:53.170236 8597 init.cc:377] Logged in from keytab as oryx/127.0.249.65@KRBTEST.COM (short username oryx)
I20260925 00:48:53.171056 8597 webserver.cc:460] Webserver started at http://127.0.249.65:46559/ using document root <none> and password file <none>
I20260925 00:48:53.171589 8597 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:53.171650 8597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:53.171846 8597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:53.173890 8597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "3a5481de9f0a4087b81b84910652c04e"
format_stamp: "Formatted at 2026-09-25 00:48:53 on dist-test-slave-05m0"
server_key: "d2b4ea1ec804e405fb65dd6d9d3ec29f"
server_key_iv: "44dca4acc1dec927359a0f08004a626e"
server_key_version: "encryptionkey@0"
I20260925 00:48:53.174495 8597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "3a5481de9f0a4087b81b84910652c04e"
format_stamp: "Formatted at 2026-09-25 00:48:53 on dist-test-slave-05m0"
server_key: "d2b4ea1ec804e405fb65dd6d9d3ec29f"
server_key_iv: "44dca4acc1dec927359a0f08004a626e"
server_key_version: "encryptionkey@0"
I20260925 00:48:53.178228 8597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:48:53.180445 8612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:53.181615 8597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:53.181742 8597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "3a5481de9f0a4087b81b84910652c04e"
format_stamp: "Formatted at 2026-09-25 00:48:53 on dist-test-slave-05m0"
server_key: "d2b4ea1ec804e405fb65dd6d9d3ec29f"
server_key_iv: "44dca4acc1dec927359a0f08004a626e"
server_key_version: "encryptionkey@0"
I20260925 00:48:53.181828 8597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:53.210224 8597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:53.214473 8597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:53.214684 8597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:53.216773 8597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:53.216845 8597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:53.216923 8597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:53.216960 8597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297333, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:53.228538 8597 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:44535
I20260925 00:48:53.228605 8727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:44535 every 8 connection(s)
I20260925 00:48:53.229759 8597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:53.232425 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.219025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:41869 (local address 127.0.249.126:44529)
0925 00:48:53.219176 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.219182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.219200 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:48:53.219514 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.219519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.219585 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.219690 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:53.221286 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.222223 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.224225 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.224687 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.224810 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.228409 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:53.228413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:53.228417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:53.228450 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:53.230243 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.230908 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.230912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.230914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.230997 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.231329 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.231333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.231335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.231520 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:53.231633 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.232117 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.232252 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:48:53.233530 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 8597
I20260925 00:48:53.233665 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:53.233834 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.217859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44529 (local address 127.0.249.65:41869)
0925 00:48:53.218380 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.218431 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.219328 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.219892 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.219902 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.220378 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:53.221098 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.221122 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.222393 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.222412 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.224049 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.224061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.225051 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.225056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.225284 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.226062 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:53.226084 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:53.228191 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:53.230445 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.230452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.230468 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.230796 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.231089 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.231093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.231095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.231233 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.231634 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:53.231641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:53.231877 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.232075 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.232379 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":153,"threads_started":1}
I20260925 00:48:53.234047 997 external_mini_cluster.cc:1428] Setting key f89ec034e22ece2fd14ff747b714e8b5
I20260925 00:48:53.235157 8728 heartbeater.cc:347] Connected to a master server at 127.0.249.126:44529
I20260925 00:48:53.235273 8728 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:53.237867 8728 heartbeater.cc:511] Master 127.0.249.126:44529 requested a full tablet report, sending...
I20260925 00:48:53.239427 8524 ts_manager.cc:194] Registered new tserver with Master: 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535)
I20260925 00:48:53.241067 8524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:41869
I20260925 00:48:53.245515 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.237228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:44749 (local address 127.0.249.126:44529)
0925 00:48:53.237340 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.237344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.237753 (+ 409us) server_negotiation.cc:407] Connection header received
0925 00:48:53.237800 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.237804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.237857 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.237947 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:53.238593 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.239316 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.240201 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.240489 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.240589 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.241935 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:53.241942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:53.241948 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:53.241981 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:53.243768 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.244283 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.244288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.244289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.244376 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.244705 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.244708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.244709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.244870 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:53.244951 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.245244 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.245332 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":43}
I20260925 00:48:53.245621 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.237188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44529 (local address 127.0.249.65:44749)
0925 00:48:53.237534 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.237553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.237639 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.238062 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.238068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.238292 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:53.238453 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.238462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.239459 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.239461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.240086 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.240094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.240691 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.240693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.240764 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.241256 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:53.241276 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:53.241754 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:53.243908 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.243912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.243914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.244165 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.244501 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.244503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.244505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.244609 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.244968 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:53.244972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:53.245071 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.245189 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.245332 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.0.249.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab.
Entry for principal oryx/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab.
I20260925 00:48:53.264223 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:44529
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab
--principal=oryx/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:53.378686 8737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:53.378983 8737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:53.379082 8737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:53.383203 8737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:53.383279 8737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:53.383345 8737 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:48:53.383422 8737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:53.388851 8737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab
--principal=oryx/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.249.126:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:53.389909 8737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:53.390899 8737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:53.400254 8745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:53.400247 8742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:53.400455 8737 server_base.cc:1034] running on GCE node
W20260925 00:48:53.400641 8743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:53.401010 8737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:53.401664 8737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:53.402866 8737 hybrid_clock.cc:648] HybridClock initialized: now 1790297333402845 us; error 32 us; skew 500 ppm
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297333, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:53.405736 8737 init.cc:377] Logged in from keytab as oryx/127.0.249.66@KRBTEST.COM (short username oryx)
I20260925 00:48:53.406564 8737 webserver.cc:460] Webserver started at http://127.0.249.66:42063/ using document root <none> and password file <none>
I20260925 00:48:53.407097 8737 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:53.407181 8737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:53.407397 8737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:53.409382 8737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "ae19466af3024b9398142b78ec66f846"
format_stamp: "Formatted at 2026-09-25 00:48:53 on dist-test-slave-05m0"
server_key: "45311f9f4ddeb01022941bc4020bf4b2"
server_key_iv: "a1bd0b586648ff3dc977c6d8b1e5f493"
server_key_version: "encryptionkey@0"
I20260925 00:48:53.409965 8737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "ae19466af3024b9398142b78ec66f846"
format_stamp: "Formatted at 2026-09-25 00:48:53 on dist-test-slave-05m0"
server_key: "45311f9f4ddeb01022941bc4020bf4b2"
server_key_iv: "a1bd0b586648ff3dc977c6d8b1e5f493"
server_key_version: "encryptionkey@0"
I20260925 00:48:53.413470 8737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 00:48:53.415761 8752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:53.416949 8737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:48:53.417073 8737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "ae19466af3024b9398142b78ec66f846"
format_stamp: "Formatted at 2026-09-25 00:48:53 on dist-test-slave-05m0"
server_key: "45311f9f4ddeb01022941bc4020bf4b2"
server_key_iv: "a1bd0b586648ff3dc977c6d8b1e5f493"
server_key_version: "encryptionkey@0"
I20260925 00:48:53.417176 8737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:53.431632 8737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:53.434382 8737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:53.434576 8737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:53.435981 8737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:53.436049 8737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:53.436120 8737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:53.436167 8737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297333, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:53.446787 8737 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:41609
I20260925 00:48:53.446857 8867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:41609 every 8 connection(s)
I20260925 00:48:53.447687 8737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:53.449950 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 8737
I20260925 00:48:53.450093 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:53.450379 997 external_mini_cluster.cc:1428] Setting key 6f1b35b567f49a3a08be31ee2821de98
I20260925 00:48:53.450760 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.436799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:59557 (local address 127.0.249.126:44529)
0925 00:48:53.436941 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.436945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.437975 (+ 1030us) server_negotiation.cc:407] Connection header received
0925 00:48:53.438906 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.438913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.438985 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.439103 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:53.440645 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.441533 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.443281 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.443702 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.443826 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.446981 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:53.446985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:53.446987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:53.447019 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:53.448686 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.449183 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.449185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.449187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.449262 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.449609 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.449611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.449612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.449851 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:53.449959 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.450474 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.450610 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:48:53.451552 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.437340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44529 (local address 127.0.249.66:59557)
0925 00:48:53.437821 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.437861 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.438670 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.439296 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.439307 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.439809 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:53.440477 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.440499 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.441683 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.441688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.443140 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.443152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.443984 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.443989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.444191 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.444774 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:53.444799 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:53.446817 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:53.448780 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.448785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.448795 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.449108 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.449338 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.449340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.449341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.449523 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.449960 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:53.449964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:53.450233 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.450383 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.450659 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":128,"threads_started":1}
I20260925 00:48:53.452725 8868 heartbeater.cc:347] Connected to a master server at 127.0.249.126:44529
I20260925 00:48:53.452872 8868 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:53.453474 8868 heartbeater.cc:511] Master 127.0.249.126:44529 requested a full tablet report, sending...
I20260925 00:48:53.454499 8525 ts_manager.cc:194] Registered new tserver with Master: ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609)
I20260925 00:48:53.455304 8525 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:59557
WARNING: no policy specified for oryx/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:48:53.461961 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.454940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:38457 (local address 127.0.249.126:44529)
0925 00:48:53.455101 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.455105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.455321 (+ 216us) server_negotiation.cc:407] Connection header received
0925 00:48:53.455425 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.455428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.455471 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.455563 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:53.456039 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.456642 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.457556 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.457783 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.457885 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.458954 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:53.458958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:53.458961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:53.458983 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:53.460337 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.460736 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.460742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.460747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.460825 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.461098 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.461101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.461103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.461302 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:53.461423 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.461687 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.461792 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 00:48:53.461968 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.454849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44529 (local address 127.0.249.66:38457)
0925 00:48:53.455228 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.455242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.455340 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.455676 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.455679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.455841 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:53.455933 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.455938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.456768 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.456771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.457439 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.457449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.457887 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.457890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.457980 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.458516 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:53.458526 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:53.458821 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:53.460446 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.460451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.460453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.460642 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.460910 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.460914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.460919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.461006 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.461439 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:53.461443 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:53.461533 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.461636 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.461775 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab.
Entry for principal oryx/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab.
I20260925 00:48:53.478345 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:44529
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab
--principal=oryx/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:53.587584 8877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:53.587834 8877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:53.587908 8877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:53.591842 8877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:53.591902 8877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:53.591939 8877 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 00:48:53.592023 8877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:53.597901 8877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab
--principal=oryx/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.249.126:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:53.598948 8877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:53.599958 8877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:53.609704 8885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:53.609688 8883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:53.609805 8877 server_base.cc:1034] running on GCE node
W20260925 00:48:53.609691 8882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:53.610387 8877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:53.611073 8877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:53.612264 8877 hybrid_clock.cc:648] HybridClock initialized: now 1790297333612242 us; error 31 us; skew 500 ppm
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297333, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:53.615020 8877 init.cc:377] Logged in from keytab as oryx/127.0.249.67@KRBTEST.COM (short username oryx)
I20260925 00:48:53.615849 8877 webserver.cc:460] Webserver started at http://127.0.249.67:35225/ using document root <none> and password file <none>
I20260925 00:48:53.616392 8877 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:53.616500 8877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:53.616722 8877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:53.618770 8877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "4d45803d22a141c4abbb3014e5d7ee6e"
format_stamp: "Formatted at 2026-09-25 00:48:53 on dist-test-slave-05m0"
server_key: "51d09b37f78b85552207450ca414c1b6"
server_key_iv: "1a16283aee2faa9434e88d63e9d94998"
server_key_version: "encryptionkey@0"
I20260925 00:48:53.619323 8877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "4d45803d22a141c4abbb3014e5d7ee6e"
format_stamp: "Formatted at 2026-09-25 00:48:53 on dist-test-slave-05m0"
server_key: "51d09b37f78b85552207450ca414c1b6"
server_key_iv: "1a16283aee2faa9434e88d63e9d94998"
server_key_version: "encryptionkey@0"
I20260925 00:48:53.623183 8877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:48:53.625792 8892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:53.626911 8877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:53.627034 8877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "4d45803d22a141c4abbb3014e5d7ee6e"
format_stamp: "Formatted at 2026-09-25 00:48:53 on dist-test-slave-05m0"
server_key: "51d09b37f78b85552207450ca414c1b6"
server_key_iv: "1a16283aee2faa9434e88d63e9d94998"
server_key_version: "encryptionkey@0"
I20260925 00:48:53.627136 8877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:53.640962 8877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:53.644044 8877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:53.644248 8877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:53.646230 8877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:53.646302 8877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:53.646386 8877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:53.646458 8877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297333, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:53.657534 8877 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:42747
I20260925 00:48:53.658221 9007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:42747 every 8 connection(s)
I20260925 00:48:53.658485 8877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:53.660353 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.646975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:57471 (local address 127.0.249.126:44529)
0925 00:48:53.647094 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.647097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.648111 (+ 1014us) server_negotiation.cc:407] Connection header received
0925 00:48:53.649021 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.649025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.649085 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.649180 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:53.650508 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.651153 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.652473 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.652795 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.652906 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.656543 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:53.656546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:53.656549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:53.656576 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:53.658098 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.658682 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.658684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.658686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.658774 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.659182 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.659186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.659190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.659445 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:53.659558 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.660053 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.660176 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:48:53.661068 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.647480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44529 (local address 127.0.249.67:57471)
0925 00:48:53.647971 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.648012 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.648813 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.649362 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.649371 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.649911 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:53.650357 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.650370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.651280 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.651283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.652358 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.652367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.652965 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.652969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.653192 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.653845 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:53.653861 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:53.656340 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:53.658209 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.658215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.658229 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.658579 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.658865 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.658868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.658870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.659038 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.659566 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:53.659574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:53.659804 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.659991 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.660298 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":141,"threads_started":1}
I20260925 00:48:53.662233 9008 heartbeater.cc:347] Connected to a master server at 127.0.249.126:44529
I20260925 00:48:53.662381 9008 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:53.662832 9008 heartbeater.cc:511] Master 127.0.249.126:44529 requested a full tablet report, sending...
I20260925 00:48:53.663887 8524 ts_manager.cc:194] Registered new tserver with Master: 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67:42747)
I20260925 00:48:53.664383 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 8877
I20260925 00:48:53.664537 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:53.664670 8524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:57471
I20260925 00:48:53.664870 997 external_mini_cluster.cc:1428] Setting key 7bfab11ddda1af7f082d6f268e3eeb9c
I20260925 00:48:53.666937 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:48:53.672114 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.664466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:43273 (local address 127.0.249.126:44529)
0925 00:48:53.665057 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.665061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.665078 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:48:53.665105 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.665107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.665137 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.665229 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:53.665933 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.666699 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.667525 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.667898 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.668016 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.669189 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:53.669191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:53.669193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:53.669212 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:53.670579 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.670998 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.671002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.671004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.671083 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.671348 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.671353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.671354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.671524 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:53.671625 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.671864 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.671963 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522}
I20260925 00:48:53.672128 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.664536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44529 (local address 127.0.249.67:43273)
0925 00:48:53.664817 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.664832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.664931 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.665247 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.665250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.665497 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:53.665603 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.665611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.666818 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.666821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.667416 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.667424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.668023 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.668025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.668092 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.668602 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:53.668613 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:53.669066 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:53.670693 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.670697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.670699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.670900 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.671176 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:53.671179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:53.671181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:53.671263 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:53.671617 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:53.671621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:53.671712 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.671813 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.671963 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 00:48:53.691061 8574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:53.685141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.0.249.126:44529)
0925 00:48:53.685254 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.685258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.685270 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:53.685306 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.685308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.685344 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.685490 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:53.686290 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.686793 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.687614 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.687813 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.687885 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.690888 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41430: BlockingRecv error: recv got EOF from 127.0.0.1:41430 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 00:48:53 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:53.700124 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.692301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.0.249.126:44529)
0925 00:48:53.692437 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.692441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.692554 (+ 113us) server_negotiation.cc:407] Connection header received
0925 00:48:53.692694 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.692697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.692736 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.692816 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:53.693436 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.693924 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.694735 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.694963 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.695059 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.697216 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:53.697220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:53.697223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:53.697247 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:53.698509 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.698921 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.698925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.698927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.699014 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:53.699315 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:53.699317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:53.699318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:53.699480 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:53.699601 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.699814 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.699940 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:48:53.702818 8524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:53.705797 8524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:53.719434 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.713669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:44535 (local address 127.0.0.1:50462)
0925 00:48:53.714180 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.714229 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.714346 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.715357 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.715363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.715382 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:53.715536 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.715544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.716701 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.716707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.718004 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.718013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.718920 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.718924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.719027 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.719068 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.719175 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.719286 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":39,"thread_start_us":69,"threads_started":1}
I20260925 00:48:53.719470 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.713813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.0.249.65:44535)
0925 00:48:53.714854 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.714870 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.714894 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:48:53.715012 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.715017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.715227 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.715345 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:53.715645 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.716600 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.718129 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.718806 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.718926 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.719085 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.719218 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.719325 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":75,"threads_started":1}
I20260925 00:48:53.720189 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.714336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:41609 (local address 127.0.0.1:34508)
0925 00:48:53.715059 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.715077 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.715186 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.716093 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.716096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.716114 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:53.716207 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.716212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.717623 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.717627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.718686 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.718693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.719733 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.719736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.719835 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.719867 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.719965 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.720038 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":18,"thread_start_us":94,"threads_started":1}
I20260925 00:48:53.720242 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.714405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.249.67:42747)
0925 00:48:53.714728 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.714739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.715012 (+ 273us) server_negotiation.cc:407] Connection header received
0925 00:48:53.715213 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.715221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.715426 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.715578 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:53.715805 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.716961 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.718475 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.719258 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.719355 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.719507 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.720019 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.720107 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":118,"threads_started":1}
I20260925 00:48:53.720582 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.714715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.0.249.66:41609)
0925 00:48:53.714993 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.715005 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.715671 (+ 666us) server_negotiation.cc:407] Connection header received
0925 00:48:53.715771 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.715777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.715954 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.716104 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:53.716317 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.717487 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.718791 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.719613 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.719728 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.719881 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.720354 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.720449 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
I20260925 00:48:53.720592 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.713847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42747 (local address 127.0.0.1:51860)
0925 00:48:53.714920 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.714939 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.715076 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.715559 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.715561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.715576 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:53.715694 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.715700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.717105 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.717108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.718337 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.718347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.719641 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.719644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.719796 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.719823 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.720414 (+ 591us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.720481 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260925 00:48:53.723069 8942 tablet_service.cc:1467] Processing CreateTablet for tablet 97624b21665f49e38a0b3d169fb8ef52 (DEFAULT_TABLE table=test-table [id=14cb255b86604cd7a5ea06042d46137f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:53.723067 8662 tablet_service.cc:1467] Processing CreateTablet for tablet 97624b21665f49e38a0b3d169fb8ef52 (DEFAULT_TABLE table=test-table [id=14cb255b86604cd7a5ea06042d46137f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:53.723165 8784 tablet_service.cc:1467] Processing CreateTablet for tablet 97624b21665f49e38a0b3d169fb8ef52 (DEFAULT_TABLE table=test-table [id=14cb255b86604cd7a5ea06042d46137f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:53.724131 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97624b21665f49e38a0b3d169fb8ef52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:53.724144 8942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97624b21665f49e38a0b3d169fb8ef52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:53.724144 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97624b21665f49e38a0b3d169fb8ef52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:53.730957 9036 tablet_bootstrap.cc:492] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: Bootstrap starting.
I20260925 00:48:53.732664 9038 tablet_bootstrap.cc:492] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: Bootstrap starting.
I20260925 00:48:53.733371 9036 tablet_bootstrap.cc:654] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:53.733709 9037 tablet_bootstrap.cc:492] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: Bootstrap starting.
I20260925 00:48:53.734282 9036 log.cc:824] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:53.734794 9038 tablet_bootstrap.cc:654] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:53.735639 9038 log.cc:824] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:53.735740 9037 tablet_bootstrap.cc:654] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:53.736518 9037 log.cc:824] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:53.736857 9036 tablet_bootstrap.cc:492] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: No bootstrap required, opened a new log
I20260925 00:48:53.737044 9036 ts_tablet_manager.cc:1397] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20260925 00:48:53.737509 9038 tablet_bootstrap.cc:492] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: No bootstrap required, opened a new log
I20260925 00:48:53.737700 9038 ts_tablet_manager.cc:1397] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260925 00:48:53.738278 9037 tablet_bootstrap.cc:492] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: No bootstrap required, opened a new log
I20260925 00:48:53.738451 9037 ts_tablet_manager.cc:1397] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:48:53.740072 9036 raft_consensus.cc:348] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } }
I20260925 00:48:53.740301 9036 raft_consensus.cc:374] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:53.740353 9036 raft_consensus.cc:729] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae19466af3024b9398142b78ec66f846, State: Initialized, Role: FOLLOWER
I20260925 00:48:53.740787 9036 consensus_queue.cc:260] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } }
I20260925 00:48:53.741436 8868 heartbeater.cc:503] Master 127.0.249.126:44529 was elected leader, sending a full tablet report...
I20260925 00:48:53.742154 9036 ts_tablet_manager.cc:1428] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:48:53.742388 9037 raft_consensus.cc:348] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } }
I20260925 00:48:53.742637 9037 raft_consensus.cc:374] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:53.742703 9037 raft_consensus.cc:729] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5481de9f0a4087b81b84910652c04e, State: Initialized, Role: FOLLOWER
I20260925 00:48:53.742539 9038 raft_consensus.cc:348] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } }
I20260925 00:48:53.742795 9038 raft_consensus.cc:374] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:53.742852 9038 raft_consensus.cc:729] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d45803d22a141c4abbb3014e5d7ee6e, State: Initialized, Role: FOLLOWER
I20260925 00:48:53.743136 9037 consensus_queue.cc:260] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } }
I20260925 00:48:53.743188 9038 consensus_queue.cc:260] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } }
I20260925 00:48:53.743975 8728 heartbeater.cc:503] Master 127.0.249.126:44529 was elected leader, sending a full tablet report...
I20260925 00:48:53.744029 9008 heartbeater.cc:503] Master 127.0.249.126:44529 was elected leader, sending a full tablet report...
I20260925 00:48:53.744087 9038 ts_tablet_manager.cc:1428] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260925 00:48:53.745149 9037 ts_tablet_manager.cc:1428] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 00:48:53.885746 9044 raft_consensus.cc:482] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:53.885960 9044 raft_consensus.cc:504] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } }
I20260925 00:48:53.887127 9044 leader_election.cc:288] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535), ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609)
I20260925 00:48:53.891546 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.887349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:44535 (local address 127.0.249.67:43969)
0925 00:48:53.887477 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.887500 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.887607 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.887975 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.887978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.887998 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:53.888140 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.888149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.889145 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.889147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.890476 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.890487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.891184 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.891187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.891274 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.891296 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.891356 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.891421 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 00:48:53.891597 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.887436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:43969 (local address 127.0.249.65:44535)
0925 00:48:53.887569 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.887573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.887588 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:48:53.887730 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.887737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.887791 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.887882 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:53.888255 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.889040 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.890598 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.891062 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.891174 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.891223 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.891394 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.891459 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:48:53.892025 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.887674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:41609 (local address 127.0.249.67:35119)
0925 00:48:53.888002 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:53.888015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:53.888108 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:53.888352 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:53.888354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:53.888370 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:53.888470 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.888475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.889342 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.889344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.890784 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:53.890791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.891573 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.891576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:53.891674 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.891699 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:48:53.891820 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 00:48:53.891886 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260925 00:48:53.892016 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.887829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:35119 (local address 127.0.249.66:41609)
0925 00:48:53.887948 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.887953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.888102 (+ 149us) server_negotiation.cc:407] Connection header received
0925 00:48:53.888185 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.888187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.888259 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.888336 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:53.888576 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.889250 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.890885 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.891446 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.891567 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.891610 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.891843 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.891913 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:48:53.892195 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97624b21665f49e38a0b3d169fb8ef52" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5481de9f0a4087b81b84910652c04e" is_pre_election: true
I20260925 00:48:53.892525 8682 raft_consensus.cc:2454] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 0.
I20260925 00:48:53.892623 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97624b21665f49e38a0b3d169fb8ef52" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19466af3024b9398142b78ec66f846" is_pre_election: true
I20260925 00:48:53.892913 8809 raft_consensus.cc:2454] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 0.
I20260925 00:48:53.892997 8893 leader_election.cc:302] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5481de9f0a4087b81b84910652c04e, 4d45803d22a141c4abbb3014e5d7ee6e; no voters:
I20260925 00:48:53.893323 9044 raft_consensus.cc:2781] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:53.893446 9044 raft_consensus.cc:482] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:53.893503 9044 raft_consensus.cc:3037] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:53.894521 9044 raft_consensus.cc:504] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } }
I20260925 00:48:53.894879 9044 leader_election.cc:288] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 election: Requested vote from peers 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535), ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609)
I20260925 00:48:53.895159 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97624b21665f49e38a0b3d169fb8ef52" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5481de9f0a4087b81b84910652c04e"
I20260925 00:48:53.895252 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97624b21665f49e38a0b3d169fb8ef52" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19466af3024b9398142b78ec66f846"
I20260925 00:48:53.895292 8682 raft_consensus.cc:3037] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:53.895364 8809 raft_consensus.cc:3037] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:53.896557 8809 raft_consensus.cc:2454] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 1.
I20260925 00:48:53.896557 8682 raft_consensus.cc:2454] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 1.
I20260925 00:48:53.896834 8893 leader_election.cc:302] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d45803d22a141c4abbb3014e5d7ee6e, ae19466af3024b9398142b78ec66f846; no voters:
I20260925 00:48:53.897015 9044 raft_consensus.cc:2781] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:53.897246 9044 raft_consensus.cc:686] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 LEADER]: Becoming Leader. State: Replica: 4d45803d22a141c4abbb3014e5d7ee6e, State: Running, Role: LEADER
I20260925 00:48:53.897598 9044 consensus_queue.cc:237] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } }
I20260925 00:48:53.900625 8525 catalog_manager.cc:5495] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67). New cstate: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } health_report { overall_health: UNKNOWN } } }
W20260925 00:48:53.909509 9009 tablet.cc:2369] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:48:53.949666 8869 tablet.cc:2369] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:53.952085 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:53.948753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.0.249.67:42747)
0925 00:48:53.948887 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:53.948890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:53.948921 (+ 31us) server_negotiation.cc:407] Connection header received
0925 00:48:53.948994 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:53.948996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:53.949051 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:53.949135 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:53.949353 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.949970 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.950835 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:53.951062 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:53.951139 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:53.951352 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:53.951760 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:53.951844 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:53.951897 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 00:48:53.951980 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:48:53.957726 8809 raft_consensus.cc:1261] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Refusing update from remote peer 4d45803d22a141c4abbb3014e5d7ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:53.957762 8682 raft_consensus.cc:1261] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Refusing update from remote peer 4d45803d22a141c4abbb3014e5d7ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:53.958398 9044 consensus_queue.cc:1035] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:53.958595 9046 consensus_queue.cc:1035] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:53.966501 9050 mvcc.cc:205] Tried to move back new op lower bound from 7333057879883980800 to 7333057879647219712. Current Snapshot: MvccSnapshot[applied={T|T < 7333057879883980800}]
I20260925 00:48:53.966902 9051 mvcc.cc:205] Tried to move back new op lower bound from 7333057879883980800 to 7333057879647219712. Current Snapshot: MvccSnapshot[applied={T|T < 7333057879883980800}]
I20260925 00:48:53.973872 8523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:53.974004 8523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:53.976240 8523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: Sending DeleteTablet for 3 replicas of tablet 97624b21665f49e38a0b3d169fb8ef52
I20260925 00:48:53.976962 8942 tablet_service.cc:1514] Processing DeleteTablet for tablet 97624b21665f49e38a0b3d169fb8ef52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:53 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:51860
I20260925 00:48:53.977005 8662 tablet_service.cc:1514] Processing DeleteTablet for tablet 97624b21665f49e38a0b3d169fb8ef52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:53 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:50462
I20260925 00:48:53.977475 9058 tablet_replica.cc:330] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: stopping tablet replica
I20260925 00:48:53.977586 8784 tablet_service.cc:1514] Processing DeleteTablet for tablet 97624b21665f49e38a0b3d169fb8ef52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:53 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:34508
I20260925 00:48:53.977602 8523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 00:48:53.977773 9058 raft_consensus.cc:2229] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 LEADER]: Raft consensus shutting down.
W20260925 00:48:53.977960 8523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:53.977982 9060 tablet_replica.cc:330] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: stopping tablet replica
I20260925 00:48:53.978081 9058 raft_consensus.cc:2258] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:53.978241 9060 raft_consensus.cc:2229] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:53.978436 9059 tablet_replica.cc:330] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: stopping tablet replica
I20260925 00:48:53.978519 9060 raft_consensus.cc:2258] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:53.978757 9059 raft_consensus.cc:2229] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:53.979005 9059 raft_consensus.cc:2258] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:53.980464 9059 ts_tablet_manager.cc:1905] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:53.980464 9058 ts_tablet_manager.cc:1905] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:53.983254 9058 ts_tablet_manager.cc:1918] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:53.983280 8662 tablet_service.cc:1467] Processing CreateTablet for tablet 0a164fc0ec014459b78ba99ee5cc61ba (DEFAULT_TABLE table=test-table [id=536022b7fdb644d19c7a12561d3dede3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:53.983353 9058 log.cc:1196] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/97624b21665f49e38a0b3d169fb8ef52
I20260925 00:48:53.983536 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a164fc0ec014459b78ba99ee5cc61ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:53.983675 9058 ts_tablet_manager.cc:1939] T 97624b21665f49e38a0b3d169fb8ef52 P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting consensus metadata
I20260925 00:48:53.984027 9059 ts_tablet_manager.cc:1918] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:53.984167 9059 log.cc:1196] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/97624b21665f49e38a0b3d169fb8ef52
I20260925 00:48:53.984511 9059 ts_tablet_manager.cc:1939] T 97624b21665f49e38a0b3d169fb8ef52 P 3a5481de9f0a4087b81b84910652c04e: Deleting consensus metadata
I20260925 00:48:53.985622 8509 catalog_manager.cc:4841] TS 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535): tablet 97624b21665f49e38a0b3d169fb8ef52 (table test-table [id=14cb255b86604cd7a5ea06042d46137f]) successfully deleted
I20260925 00:48:53.985666 8784 tablet_service.cc:1467] Processing CreateTablet for tablet 0a164fc0ec014459b78ba99ee5cc61ba (DEFAULT_TABLE table=test-table [id=536022b7fdb644d19c7a12561d3dede3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:53.985922 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a164fc0ec014459b78ba99ee5cc61ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:53.988183 9036 tablet_bootstrap.cc:492] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: Bootstrap starting.
I20260925 00:48:53.988308 8512 catalog_manager.cc:4841] TS 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67:42747): tablet 97624b21665f49e38a0b3d169fb8ef52 (table test-table [id=14cb255b86604cd7a5ea06042d46137f]) successfully deleted
I20260925 00:48:53.988775 8942 tablet_service.cc:1467] Processing CreateTablet for tablet 0a164fc0ec014459b78ba99ee5cc61ba (DEFAULT_TABLE table=test-table [id=536022b7fdb644d19c7a12561d3dede3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:53.988878 9037 tablet_bootstrap.cc:492] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: Bootstrap starting.
I20260925 00:48:53.989033 8942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a164fc0ec014459b78ba99ee5cc61ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:53.989040 9036 tablet_bootstrap.cc:654] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:53.989883 9037 tablet_bootstrap.cc:654] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:53.990737 9036 tablet_bootstrap.cc:492] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: No bootstrap required, opened a new log
I20260925 00:48:53.990814 9036 ts_tablet_manager.cc:1397] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 00:48:53.990999 9038 tablet_bootstrap.cc:492] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: Bootstrap starting.
I20260925 00:48:53.991343 9036 raft_consensus.cc:348] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:53.991461 9036 raft_consensus.cc:374] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:53.991680 9036 raft_consensus.cc:729] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae19466af3024b9398142b78ec66f846, State: Initialized, Role: FOLLOWER
I20260925 00:48:53.991839 9038 tablet_bootstrap.cc:654] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:53.991854 9036 consensus_queue.cc:260] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:53.991941 9060 ts_tablet_manager.cc:1905] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:53.992151 9036 ts_tablet_manager.cc:1428] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 00:48:53.993070 9038 tablet_bootstrap.cc:492] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: No bootstrap required, opened a new log
I20260925 00:48:53.993160 9038 ts_tablet_manager.cc:1397] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 00:48:53.993649 9038 raft_consensus.cc:348] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:53.993750 9038 raft_consensus.cc:374] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:53.993780 9038 raft_consensus.cc:729] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d45803d22a141c4abbb3014e5d7ee6e, State: Initialized, Role: FOLLOWER
I20260925 00:48:53.993885 9038 consensus_queue.cc:260] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:53.994138 9038 ts_tablet_manager.cc:1428] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:53.994436 9060 ts_tablet_manager.cc:1918] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:53.994514 9060 log.cc:1196] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/97624b21665f49e38a0b3d169fb8ef52
I20260925 00:48:53.995345 9060 ts_tablet_manager.cc:1939] T 97624b21665f49e38a0b3d169fb8ef52 P ae19466af3024b9398142b78ec66f846: Deleting consensus metadata
I20260925 00:48:53.996140 9037 tablet_bootstrap.cc:492] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: No bootstrap required, opened a new log
I20260925 00:48:53.996232 9037 ts_tablet_manager.cc:1397] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 00:48:53.996531 8509 catalog_manager.cc:4841] TS ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609): tablet 97624b21665f49e38a0b3d169fb8ef52 (table test-table [id=14cb255b86604cd7a5ea06042d46137f]) successfully deleted
I20260925 00:48:53.996727 9037 raft_consensus.cc:348] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:53.996833 9037 raft_consensus.cc:374] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:53.996888 9037 raft_consensus.cc:729] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5481de9f0a4087b81b84910652c04e, State: Initialized, Role: FOLLOWER
I20260925 00:48:53.997027 9037 consensus_queue.cc:260] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:53.997278 9037 ts_tablet_manager.cc:1428] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:54.085428 9042 raft_consensus.cc:482] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:54.085657 9042 raft_consensus.cc:504] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.087040 9042 leader_election.cc:288] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535), 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67:42747)
I20260925 00:48:54.091617 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.087592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:55469 (local address 127.0.249.65:44535)
0925 00:48:54.087713 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.087718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.087737 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:48:54.087883 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.087889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.087950 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.088028 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:54.088329 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.088918 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.090339 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.090990 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.091122 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.091176 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.091407 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.091484 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:48:54.091943 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.087338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:51371 (local address 127.0.249.67:42747)
0925 00:48:54.087471 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.087476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.087941 (+ 465us) server_negotiation.cc:407] Connection header received
0925 00:48:54.088105 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.088111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.088160 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.088207 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:54.088594 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.089312 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.090638 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.091174 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.091283 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.091348 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.091733 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.091820 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:48:54.092100 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.087254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:44535 (local address 127.0.249.66:55469)
0925 00:48:54.087617 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.087633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.087742 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.088043 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.088046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.088067 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:54.088199 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.088207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.089026 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.089031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.090202 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.090212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.091111 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.091115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.091241 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.091272 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.091839 (+ 567us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.091915 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":108,"threads_started":1}
I20260925 00:48:54.092293 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.087254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42747 (local address 127.0.249.66:51371)
0925 00:48:54.087829 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.087845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.087975 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.088334 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.088337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.088354 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:54.088465 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.088472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.089449 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.089453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.090535 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.090542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.091450 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.091453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.091586 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.091608 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.092065 (+ 457us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.092142 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":235,"thread_start_us":101,"threads_started":1}
I20260925 00:48:54.092525 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a164fc0ec014459b78ba99ee5cc61ba" candidate_uuid: "ae19466af3024b9398142b78ec66f846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5481de9f0a4087b81b84910652c04e" is_pre_election: true
I20260925 00:48:54.092692 8682 raft_consensus.cc:2454] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae19466af3024b9398142b78ec66f846 in term 0.
I20260925 00:48:54.092819 8962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a164fc0ec014459b78ba99ee5cc61ba" candidate_uuid: "ae19466af3024b9398142b78ec66f846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" is_pre_election: true
I20260925 00:48:54.093044 8962 raft_consensus.cc:2454] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae19466af3024b9398142b78ec66f846 in term 0.
I20260925 00:48:54.093044 8753 leader_election.cc:302] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5481de9f0a4087b81b84910652c04e, ae19466af3024b9398142b78ec66f846; no voters:
I20260925 00:48:54.093272 9042 raft_consensus.cc:2781] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:54.093338 9042 raft_consensus.cc:482] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:54.093408 9042 raft_consensus.cc:3037] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.094352 9042 raft_consensus.cc:504] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.094718 9042 leader_election.cc:288] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [CANDIDATE]: Term 1 election: Requested vote from peers 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535), 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67:42747)
I20260925 00:48:54.095029 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a164fc0ec014459b78ba99ee5cc61ba" candidate_uuid: "ae19466af3024b9398142b78ec66f846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5481de9f0a4087b81b84910652c04e"
I20260925 00:48:54.095127 8682 raft_consensus.cc:3037] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.095124 8962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a164fc0ec014459b78ba99ee5cc61ba" candidate_uuid: "ae19466af3024b9398142b78ec66f846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d45803d22a141c4abbb3014e5d7ee6e"
I20260925 00:48:54.095237 8962 raft_consensus.cc:3037] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.095914 8682 raft_consensus.cc:2454] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae19466af3024b9398142b78ec66f846 in term 1.
I20260925 00:48:54.096206 8753 leader_election.cc:302] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5481de9f0a4087b81b84910652c04e, ae19466af3024b9398142b78ec66f846; no voters:
I20260925 00:48:54.096376 9042 raft_consensus.cc:2781] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:54.096450 8962 raft_consensus.cc:2454] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae19466af3024b9398142b78ec66f846 in term 1.
I20260925 00:48:54.096594 9042 raft_consensus.cc:686] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 1 LEADER]: Becoming Leader. State: Replica: ae19466af3024b9398142b78ec66f846, State: Running, Role: LEADER
I20260925 00:48:54.096845 9042 consensus_queue.cc:237] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.099009 8523 catalog_manager.cc:5495] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae19466af3024b9398142b78ec66f846 (127.0.249.66). New cstate: current_term: 1 leader_uuid: "ae19466af3024b9398142b78ec66f846" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } health_report { overall_health: UNKNOWN } } }
Sep 25 00:48:54 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297332, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.126@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:54.151298 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.143198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.0.249.126:44529)
0925 00:48:54.143329 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.143333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.143473 (+ 140us) server_negotiation.cc:407] Connection header received
0925 00:48:54.143576 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.143578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.143618 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.143696 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:54.144308 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.144823 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.145622 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.145819 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.145894 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.148336 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:54.148340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:54.148347 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:54.148366 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:54.149633 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:54.150067 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:54.150073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:54.150075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:54.150144 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:54.150408 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:54.150411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:54.150412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:54.150583 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:54.150681 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.150962 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.151161 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":53}
I20260925 00:48:54.151298 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.143099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44529 (local address 127.0.0.1:41448)
0925 00:48:54.143371 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.143387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.143508 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.143728 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.143730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.144115 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:54.144209 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.144215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.144911 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.144913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.145522 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.145531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.145910 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.145912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.145969 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.146539 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:54.146551 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:54.148211 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:54.149727 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:54.149740 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:54.149746 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:54.149969 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:54.150238 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:54.150241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:54.150243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:54.150322 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:54.150695 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:54.150704 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:54.150808 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.150916 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.151097 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
I20260925 00:48:54.155283 8523 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:41448:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 00:48:54.155992 8523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:54.160540 8662 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5a83a149294a659986d3490f397f74 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=17758a6d799c4a64a71418a3445012d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:48:54.160641 8942 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5a83a149294a659986d3490f397f74 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=17758a6d799c4a64a71418a3445012d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:48:54.160758 8784 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5a83a149294a659986d3490f397f74 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=17758a6d799c4a64a71418a3445012d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 00:48:54.160953 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5a83a149294a659986d3490f397f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:54.160984 8942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5a83a149294a659986d3490f397f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:54.161171 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5a83a149294a659986d3490f397f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:54.163816 9036 tablet_bootstrap.cc:492] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: Bootstrap starting.
I20260925 00:48:54.163864 9037 tablet_bootstrap.cc:492] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: Bootstrap starting.
I20260925 00:48:54.164335 9038 tablet_bootstrap.cc:492] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: Bootstrap starting.
I20260925 00:48:54.164873 9037 tablet_bootstrap.cc:654] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:54.164893 9036 tablet_bootstrap.cc:654] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:54.165269 9038 tablet_bootstrap.cc:654] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:54.166178 9037 tablet_bootstrap.cc:492] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: No bootstrap required, opened a new log
I20260925 00:48:54.166271 9037 ts_tablet_manager.cc:1397] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 00:48:54.166473 9038 tablet_bootstrap.cc:492] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: No bootstrap required, opened a new log
I20260925 00:48:54.166558 9038 ts_tablet_manager.cc:1397] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:54.166713 9037 raft_consensus.cc:348] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.166831 9037 raft_consensus.cc:374] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:54.166872 9037 raft_consensus.cc:729] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5481de9f0a4087b81b84910652c04e, State: Initialized, Role: FOLLOWER
I20260925 00:48:54.166929 9038 raft_consensus.cc:348] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.167037 9038 raft_consensus.cc:374] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:54.167012 9037 consensus_queue.cc:260] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.167076 9038 raft_consensus.cc:729] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d45803d22a141c4abbb3014e5d7ee6e, State: Initialized, Role: FOLLOWER
I20260925 00:48:54.167191 9038 consensus_queue.cc:260] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.167280 9043 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.167425 9043 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:54.167480 9038 ts_tablet_manager.cc:1428] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:54.167299 9037 ts_tablet_manager.cc:1428] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:48:54.167665 9044 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.167784 9044 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:54.168943 9036 tablet_bootstrap.cc:492] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: No bootstrap required, opened a new log
I20260925 00:48:54.169039 9036 ts_tablet_manager.cc:1397] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 00:48:54.169530 9036 raft_consensus.cc:348] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.169667 9036 raft_consensus.cc:374] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:54.169714 9036 raft_consensus.cc:729] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae19466af3024b9398142b78ec66f846, State: Initialized, Role: FOLLOWER
I20260925 00:48:54.169836 9036 consensus_queue.cc:260] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.170074 9042 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.170209 9042 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:54.170694 9036 ts_tablet_manager.cc:1428] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260925 00:48:54.231228 8729 tablet.cc:2369] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:48:54.410053 9044 raft_consensus.cc:482] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:54.410172 9044 raft_consensus.cc:504] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.410542 9044 leader_election.cc:288] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535), ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609)
I20260925 00:48:54.411039 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5a83a149294a659986d3490f397f74" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5481de9f0a4087b81b84910652c04e" is_pre_election: true
I20260925 00:48:54.411058 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5a83a149294a659986d3490f397f74" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19466af3024b9398142b78ec66f846" is_pre_election: true
I20260925 00:48:54.411170 8682 raft_consensus.cc:2454] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 0.
I20260925 00:48:54.411212 8809 raft_consensus.cc:2454] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 0.
I20260925 00:48:54.411475 8893 leader_election.cc:302] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5481de9f0a4087b81b84910652c04e, 4d45803d22a141c4abbb3014e5d7ee6e; no voters:
I20260925 00:48:54.411665 9044 raft_consensus.cc:2781] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:54.411732 9044 raft_consensus.cc:482] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:54.411787 9044 raft_consensus.cc:3037] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.412775 9044 raft_consensus.cc:504] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.413089 9044 leader_election.cc:288] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 election: Requested vote from peers 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535), ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609)
I20260925 00:48:54.413477 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5a83a149294a659986d3490f397f74" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5481de9f0a4087b81b84910652c04e"
I20260925 00:48:54.413501 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5a83a149294a659986d3490f397f74" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19466af3024b9398142b78ec66f846"
I20260925 00:48:54.413590 8682 raft_consensus.cc:3037] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.413622 8809 raft_consensus.cc:3037] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.414460 8809 raft_consensus.cc:2454] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 1.
I20260925 00:48:54.414460 8682 raft_consensus.cc:2454] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 1.
I20260925 00:48:54.414749 8893 leader_election.cc:302] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5481de9f0a4087b81b84910652c04e, 4d45803d22a141c4abbb3014e5d7ee6e; no voters:
I20260925 00:48:54.414912 9044 raft_consensus.cc:2781] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:54.415023 9044 raft_consensus.cc:686] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 LEADER]: Becoming Leader. State: Replica: 4d45803d22a141c4abbb3014e5d7ee6e, State: Running, Role: LEADER
I20260925 00:48:54.415189 9044 consensus_queue.cc:237] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.415768 9046 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: TxnStatusTablet state changed. Reason: New leader 4d45803d22a141c4abbb3014e5d7ee6e. Latest consensus state: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.415884 9046 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:48:54.416097 9078 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 00:48:54.416261 9078 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 00:48:54.416857 8525 catalog_manager.cc:5495] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67). New cstate: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } health_report { overall_health: HEALTHY } } }
I20260925 00:48:54.464658 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.460416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.0.249.67:42747)
0925 00:48:54.460540 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.460545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.460597 (+ 52us) server_negotiation.cc:407] Connection header received
0925 00:48:54.460779 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.460782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.460839 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.460928 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:54.461186 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.462077 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.463366 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.463771 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.463879 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.464175 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.464260 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.464330 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.464445 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.464502 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:48:54.464656 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.460297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42747 (local address 127.0.0.1:51882)
0925 00:48:54.460537 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.460552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.460659 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.460960 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.460963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.460970 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:54.461079 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.461085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.462191 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.462195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.463212 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.463224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.463881 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.463885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.463986 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.464039 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.464344 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.464357 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.464423 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.464508 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260925 00:48:54.467689 8809 raft_consensus.cc:1261] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Refusing update from remote peer 4d45803d22a141c4abbb3014e5d7ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:54.467994 8682 raft_consensus.cc:1261] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Refusing update from remote peer 4d45803d22a141c4abbb3014e5d7ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:54.468266 9046 consensus_queue.cc:1035] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:54.468459 9044 consensus_queue.cc:1035] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:54.469792 9042 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: TxnStatusTablet state changed. Reason: New leader 4d45803d22a141c4abbb3014e5d7ee6e. Latest consensus state: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.469957 9042 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:54.470767 9043 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: TxnStatusTablet state changed. Reason: New leader 4d45803d22a141c4abbb3014e5d7ee6e. Latest consensus state: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.470912 9043 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:54.472543 9046 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.472646 9046 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:48:54.473809 9044 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.473918 9044 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P 4d45803d22a141c4abbb3014e5d7ee6e: This TxnStatusTablet replica's current role is: LEADER
I20260925 00:48:54.474763 9043 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.474869 9043 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P 3a5481de9f0a4087b81b84910652c04e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:54.475272 9081 mvcc.cc:205] Tried to move back new op lower bound from 7333057881975173120 to 7333057881765765120. Current Snapshot: MvccSnapshot[applied={T|T < 7333057881975173120 or (T in {7333057881975173120})}]
I20260925 00:48:54.475704 9042 tablet_replica.cc:439] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } } }
I20260925 00:48:54.475808 9042 tablet_replica.cc:442] T 9a5a83a149294a659986d3490f397f74 P ae19466af3024b9398142b78ec66f846: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 00:48:54.483266 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.478822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.0.249.66:41609)
0925 00:48:54.479323 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.479330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.479340 (+ 10us) server_negotiation.cc:407] Connection header received
0925 00:48:54.479427 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.479429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.479478 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.479575 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:54.479839 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.480386 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.481599 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.481928 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.482112 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.482380 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.482816 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.482947 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.483060 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.483116 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":109,"threads_started":1}
I20260925 00:48:54.494144 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.490465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:39145 (local address 127.0.249.67:42747)
0925 00:48:54.490616 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.490621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.490762 (+ 141us) server_negotiation.cc:407] Connection header received
0925 00:48:54.490883 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.490886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.490957 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.491020 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:54.491343 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.491873 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.493116 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.493325 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.493493 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.493756 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.493822 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.493876 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.493990 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.494042 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:48:54.494207 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.490352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42747 (local address 127.0.249.66:39145)
0925 00:48:54.490683 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.490697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.490798 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.491047 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.491049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.491057 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:54.491176 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.491183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.491990 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.491993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.492986 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.492997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.493483 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.493487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.493588 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.493638 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.493905 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.493912 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.493975 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.494054 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":129,"threads_started":1}
I20260925 00:48:54.500913 8682 raft_consensus.cc:1261] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Refusing update from remote peer ae19466af3024b9398142b78ec66f846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:54.501037 8962 raft_consensus.cc:1261] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Refusing update from remote peer ae19466af3024b9398142b78ec66f846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:54.501411 9066 consensus_queue.cc:1035] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:54.501572 9042 consensus_queue.cc:1035] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:54.524102 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.520279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:43243 (local address 127.0.249.66:41609)
0925 00:48:54.520386 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.520391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.520596 (+ 205us) server_negotiation.cc:407] Connection header received
0925 00:48:54.520706 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.520709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.520756 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.520834 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:54.521177 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.521987 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.522995 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.523326 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.523430 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.523688 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.523766 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.523830 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.523916 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.523969 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 00:48:54.524083 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.520226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:41609 (local address 127.0.249.67:43243)
0925 00:48:54.520512 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.520530 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.520618 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.520888 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.520891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.520899 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:54.521053 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.521062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.522104 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.522109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.522877 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.522889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.523440 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.523442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.523506 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.523548 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.523849 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.523854 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.523906 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.523974 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260925 00:48:54.553094 8525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:54.553263 8525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:54.555438 8525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: Sending DeleteTablet for 3 replicas of tablet 0a164fc0ec014459b78ba99ee5cc61ba
I20260925 00:48:54.556068 8662 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a164fc0ec014459b78ba99ee5cc61ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:54 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:50462
I20260925 00:48:54.556227 8942 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a164fc0ec014459b78ba99ee5cc61ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:54 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:51860
I20260925 00:48:54.556509 8784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a164fc0ec014459b78ba99ee5cc61ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:54 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:34508
I20260925 00:48:54.556538 9104 tablet_replica.cc:330] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: stopping tablet replica
I20260925 00:48:54.556649 9104 raft_consensus.cc:2229] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:54.556747 9104 raft_consensus.cc:2258] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:54.557423 9104 ts_tablet_manager.cc:1905] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:54.560456 9103 tablet_replica.cc:330] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: stopping tablet replica
I20260925 00:48:54.560673 9105 tablet_replica.cc:330] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: stopping tablet replica
I20260925 00:48:54.560851 9105 raft_consensus.cc:2229] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:54.561043 9103 raft_consensus.cc:2229] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:54.561510 9103 raft_consensus.cc:2258] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:54.562315 9103 ts_tablet_manager.cc:1905] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:54.562436 9105 raft_consensus.cc:2258] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:54.562805 9104 ts_tablet_manager.cc:1918] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:54.562891 9104 log.cc:1196] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/0a164fc0ec014459b78ba99ee5cc61ba
I20260925 00:48:54.563156 9104 ts_tablet_manager.cc:1939] T 0a164fc0ec014459b78ba99ee5cc61ba P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting consensus metadata
I20260925 00:48:54.565518 8512 catalog_manager.cc:4841] TS 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67:42747): tablet 0a164fc0ec014459b78ba99ee5cc61ba (table test-table [id=536022b7fdb644d19c7a12561d3dede3]) successfully deleted
I20260925 00:48:54.565793 9105 ts_tablet_manager.cc:1905] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:54.568584 9105 ts_tablet_manager.cc:1918] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:54.570284 9103 ts_tablet_manager.cc:1918] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:54.570422 9105 log.cc:1196] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/0a164fc0ec014459b78ba99ee5cc61ba
I20260925 00:48:54.570765 9105 ts_tablet_manager.cc:1939] T 0a164fc0ec014459b78ba99ee5cc61ba P ae19466af3024b9398142b78ec66f846: Deleting consensus metadata
I20260925 00:48:54.570866 9103 log.cc:1196] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/0a164fc0ec014459b78ba99ee5cc61ba
I20260925 00:48:54.571193 9103 ts_tablet_manager.cc:1939] T 0a164fc0ec014459b78ba99ee5cc61ba P 3a5481de9f0a4087b81b84910652c04e: Deleting consensus metadata
I20260925 00:48:54.571959 8509 catalog_manager.cc:4841] TS ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609): tablet 0a164fc0ec014459b78ba99ee5cc61ba (table test-table [id=536022b7fdb644d19c7a12561d3dede3]) successfully deleted
I20260925 00:48:54.572814 8509 catalog_manager.cc:4841] TS 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535): tablet 0a164fc0ec014459b78ba99ee5cc61ba (table test-table [id=536022b7fdb644d19c7a12561d3dede3]) successfully deleted
Sep 25 00:48:54 dist-test-slave-05m0 krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 00:48:54 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:54 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 00:48:54.594671 8574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:54.588121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.0.249.126:44529)
0925 00:48:54.588303 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.588309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.588549 (+ 240us) server_negotiation.cc:407] Connection header received
0925 00:48:54.588708 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.588710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.588754 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.588836 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:54.589610 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.590133 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.591005 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.591228 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.591333 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.594512 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41462: BlockingRecv error: recv got EOF from 127.0.0.1:41462 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 00:48:54 dist-test-slave-05m0 krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM
I20260925 00:48:54.604113 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.596146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.0.249.126:44529)
0925 00:48:54.596259 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.596263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.596457 (+ 194us) server_negotiation.cc:407] Connection header received
0925 00:48:54.596576 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.596579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.596621 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.596699 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:54.597312 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.597892 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.598645 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.598857 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.598955 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.601104 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:54.601112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:54.601114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:54.601137 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:54.602564 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:54.602962 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:54.602966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:54.602968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:54.603046 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:54.603317 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:54.603320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:54.603322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:54.603488 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:54.603617 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.603837 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.603949 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:48:54.606635 8525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:54.607081 8525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:54.611689 8662 tablet_service.cc:1467] Processing CreateTablet for tablet 0c902c4efe0e473094c62f777c152c8e (DEFAULT_TABLE table=test-table [id=646edb9ab6c64289a2fd41070865f30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:54.611718 8942 tablet_service.cc:1467] Processing CreateTablet for tablet 0c902c4efe0e473094c62f777c152c8e (DEFAULT_TABLE table=test-table [id=646edb9ab6c64289a2fd41070865f30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:54.611661 8784 tablet_service.cc:1467] Processing CreateTablet for tablet 0c902c4efe0e473094c62f777c152c8e (DEFAULT_TABLE table=test-table [id=646edb9ab6c64289a2fd41070865f30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:54.611985 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c902c4efe0e473094c62f777c152c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:54.611989 8942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c902c4efe0e473094c62f777c152c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:54.612049 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c902c4efe0e473094c62f777c152c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:54.614408 9038 tablet_bootstrap.cc:492] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: Bootstrap starting.
I20260925 00:48:54.614553 9037 tablet_bootstrap.cc:492] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: Bootstrap starting.
I20260925 00:48:54.614611 9036 tablet_bootstrap.cc:492] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: Bootstrap starting.
I20260925 00:48:54.615285 9038 tablet_bootstrap.cc:654] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:54.615360 9037 tablet_bootstrap.cc:654] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:54.615473 9036 tablet_bootstrap.cc:654] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:54.616490 9037 tablet_bootstrap.cc:492] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: No bootstrap required, opened a new log
I20260925 00:48:54.616537 9038 tablet_bootstrap.cc:492] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: No bootstrap required, opened a new log
I20260925 00:48:54.616593 9037 ts_tablet_manager.cc:1397] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 00:48:54.616611 9038 ts_tablet_manager.cc:1397] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:54.616968 9038 raft_consensus.cc:348] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.616968 9037 raft_consensus.cc:348] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.617092 9037 raft_consensus.cc:374] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:54.617092 9038 raft_consensus.cc:374] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:54.617129 9037 raft_consensus.cc:729] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5481de9f0a4087b81b84910652c04e, State: Initialized, Role: FOLLOWER
I20260925 00:48:54.617129 9038 raft_consensus.cc:729] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d45803d22a141c4abbb3014e5d7ee6e, State: Initialized, Role: FOLLOWER
I20260925 00:48:54.617244 9038 consensus_queue.cc:260] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.617244 9037 consensus_queue.cc:260] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.617558 9037 ts_tablet_manager.cc:1428] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 00:48:54.618239 9036 tablet_bootstrap.cc:492] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: No bootstrap required, opened a new log
I20260925 00:48:54.618328 9036 ts_tablet_manager.cc:1397] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 00:48:54.618691 9038 ts_tablet_manager.cc:1428] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 00:48:54.618778 9036 raft_consensus.cc:348] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.618884 9036 raft_consensus.cc:374] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:54.618913 9036 raft_consensus.cc:729] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae19466af3024b9398142b78ec66f846, State: Initialized, Role: FOLLOWER
I20260925 00:48:54.619042 9036 consensus_queue.cc:260] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.619300 9036 ts_tablet_manager.cc:1428] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:48:54.751330 9101 raft_consensus.cc:482] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:54.751545 9101 raft_consensus.cc:504] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.752094 9101 leader_election.cc:288] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535), ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609)
I20260925 00:48:54.752650 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c902c4efe0e473094c62f777c152c8e" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5481de9f0a4087b81b84910652c04e" is_pre_election: true
I20260925 00:48:54.752789 8682 raft_consensus.cc:2454] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 0.
I20260925 00:48:54.752787 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c902c4efe0e473094c62f777c152c8e" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19466af3024b9398142b78ec66f846" is_pre_election: true
I20260925 00:48:54.753001 8809 raft_consensus.cc:2454] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 0.
I20260925 00:48:54.753122 8893 leader_election.cc:302] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5481de9f0a4087b81b84910652c04e, 4d45803d22a141c4abbb3014e5d7ee6e; no voters:
I20260925 00:48:54.753296 9101 raft_consensus.cc:2781] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:54.753384 9101 raft_consensus.cc:482] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:54.753468 9101 raft_consensus.cc:3037] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.754499 9101 raft_consensus.cc:504] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.754837 9101 leader_election.cc:288] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 election: Requested vote from peers 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535), ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609)
I20260925 00:48:54.755167 8682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c902c4efe0e473094c62f777c152c8e" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5481de9f0a4087b81b84910652c04e"
I20260925 00:48:54.755210 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c902c4efe0e473094c62f777c152c8e" candidate_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19466af3024b9398142b78ec66f846"
I20260925 00:48:54.755328 8682 raft_consensus.cc:3037] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.755331 8809 raft_consensus.cc:3037] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.756183 8682 raft_consensus.cc:2454] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 1.
I20260925 00:48:54.756191 8809 raft_consensus.cc:2454] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d45803d22a141c4abbb3014e5d7ee6e in term 1.
I20260925 00:48:54.756435 8893 leader_election.cc:302] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d45803d22a141c4abbb3014e5d7ee6e, ae19466af3024b9398142b78ec66f846; no voters:
I20260925 00:48:54.756630 9101 raft_consensus.cc:2781] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:54.756726 9101 raft_consensus.cc:686] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 LEADER]: Becoming Leader. State: Replica: 4d45803d22a141c4abbb3014e5d7ee6e, State: Running, Role: LEADER
I20260925 00:48:54.756862 9101 consensus_queue.cc:237] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.758464 8525 catalog_manager.cc:5495] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67). New cstate: current_term: 1 leader_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } health_report { overall_health: HEALTHY } } }
I20260925 00:48:54.773962 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.771120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.0.249.67:42747)
0925 00:48:54.771252 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.771256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.771268 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:54.771391 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.771393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.771435 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.771497 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:54.771703 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.772172 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.772967 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.773171 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.773258 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.773565 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.773632 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.773683 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.773758 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.773798 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:48:54.775736 8682 raft_consensus.cc:1261] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Refusing update from remote peer 4d45803d22a141c4abbb3014e5d7ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:54.775833 8809 raft_consensus.cc:1261] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Refusing update from remote peer 4d45803d22a141c4abbb3014e5d7ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:54.776178 9046 consensus_queue.cc:1035] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:54.776307 9101 consensus_queue.cc:1035] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:54.783078 8523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:54.783196 8523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:54.784801 8523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: Sending DeleteTablet for 3 replicas of tablet 0c902c4efe0e473094c62f777c152c8e
I20260925 00:48:54.785362 8662 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c902c4efe0e473094c62f777c152c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:54 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:50462
I20260925 00:48:54.785430 8942 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c902c4efe0e473094c62f777c152c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:54 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:51860
I20260925 00:48:54.785578 9103 tablet_replica.cc:330] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: stopping tablet replica
I20260925 00:48:54.785589 8784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c902c4efe0e473094c62f777c152c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:54 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:34508
I20260925 00:48:54.785701 9103 raft_consensus.cc:2229] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:54.785748 9105 tablet_replica.cc:330] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: stopping tablet replica
I20260925 00:48:54.785758 9104 tablet_replica.cc:330] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: stopping tablet replica
I20260925 00:48:54.785835 9103 raft_consensus.cc:2258] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:54.785854 9104 raft_consensus.cc:2229] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:54.785852 9105 raft_consensus.cc:2229] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:54.785972 9105 raft_consensus.cc:2258] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:54.786012 9104 raft_consensus.cc:2258] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:54.786083 8523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:48:54.786473 8523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:48:54.786680 9105 ts_tablet_manager.cc:1905] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:54.786827 9103 ts_tablet_manager.cc:1905] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:54.788625 9105 ts_tablet_manager.cc:1918] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:54.788687 9105 log.cc:1196] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/0c902c4efe0e473094c62f777c152c8e
I20260925 00:48:54.788812 9103 ts_tablet_manager.cc:1918] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:54.788897 9103 log.cc:1196] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/0c902c4efe0e473094c62f777c152c8e
I20260925 00:48:54.788957 9105 ts_tablet_manager.cc:1939] T 0c902c4efe0e473094c62f777c152c8e P ae19466af3024b9398142b78ec66f846: Deleting consensus metadata
I20260925 00:48:54.789140 9103 ts_tablet_manager.cc:1939] T 0c902c4efe0e473094c62f777c152c8e P 3a5481de9f0a4087b81b84910652c04e: Deleting consensus metadata
I20260925 00:48:54.789631 8509 catalog_manager.cc:4841] TS ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609): tablet 0c902c4efe0e473094c62f777c152c8e (table test-table [id=646edb9ab6c64289a2fd41070865f30c]) successfully deleted
I20260925 00:48:54.789803 8509 catalog_manager.cc:4841] TS 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535): tablet 0c902c4efe0e473094c62f777c152c8e (table test-table [id=646edb9ab6c64289a2fd41070865f30c]) successfully deleted
I20260925 00:48:54.790380 9104 ts_tablet_manager.cc:1905] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:54.792518 9104 ts_tablet_manager.cc:1918] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:48:54.792608 9104 log.cc:1196] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/0c902c4efe0e473094c62f777c152c8e
I20260925 00:48:54.792872 9104 ts_tablet_manager.cc:1939] T 0c902c4efe0e473094c62f777c152c8e P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting consensus metadata
I20260925 00:48:54.792936 8662 tablet_service.cc:1467] Processing CreateTablet for tablet 7225d619f1924268928096ab99da59b4 (DEFAULT_TABLE table=test-table [id=846de2766b80439494878d3c8b6b453f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:54.793220 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7225d619f1924268928096ab99da59b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:54.793236 8784 tablet_service.cc:1467] Processing CreateTablet for tablet 7225d619f1924268928096ab99da59b4 (DEFAULT_TABLE table=test-table [id=846de2766b80439494878d3c8b6b453f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:54.793231 8942 tablet_service.cc:1467] Processing CreateTablet for tablet 7225d619f1924268928096ab99da59b4 (DEFAULT_TABLE table=test-table [id=846de2766b80439494878d3c8b6b453f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:48:54.793476 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7225d619f1924268928096ab99da59b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:54.793476 8942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7225d619f1924268928096ab99da59b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:54.795625 9038 tablet_bootstrap.cc:492] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: Bootstrap starting.
I20260925 00:48:54.795670 9037 tablet_bootstrap.cc:492] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e: Bootstrap starting.
I20260925 00:48:54.796525 9038 tablet_bootstrap.cc:654] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:54.796800 9037 tablet_bootstrap.cc:654] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:54.797421 9036 tablet_bootstrap.cc:492] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: Bootstrap starting.
I20260925 00:48:54.797621 9038 tablet_bootstrap.cc:492] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: No bootstrap required, opened a new log
I20260925 00:48:54.797691 9038 ts_tablet_manager.cc:1397] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:54.798075 9038 raft_consensus.cc:348] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.798197 9038 raft_consensus.cc:374] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:54.798221 9036 tablet_bootstrap.cc:654] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:54.798233 9038 raft_consensus.cc:729] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d45803d22a141c4abbb3014e5d7ee6e, State: Initialized, Role: FOLLOWER
I20260925 00:48:54.798348 9038 consensus_queue.cc:260] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.798631 9038 ts_tablet_manager.cc:1428] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 00:48:54.799275 9037 tablet_bootstrap.cc:492] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e: No bootstrap required, opened a new log
I20260925 00:48:54.799373 9037 ts_tablet_manager.cc:1397] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 00:48:54.799762 9037 raft_consensus.cc:348] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.799873 9037 raft_consensus.cc:374] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:54.799919 9037 raft_consensus.cc:729] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5481de9f0a4087b81b84910652c04e, State: Initialized, Role: FOLLOWER
I20260925 00:48:54.800051 9037 consensus_queue.cc:260] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.800297 9037 ts_tablet_manager.cc:1428] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:48:54.800783 8512 catalog_manager.cc:4841] TS 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67:42747): tablet 0c902c4efe0e473094c62f777c152c8e (table test-table [id=646edb9ab6c64289a2fd41070865f30c]) successfully deleted
I20260925 00:48:54.801752 9036 tablet_bootstrap.cc:492] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: No bootstrap required, opened a new log
I20260925 00:48:54.801842 9036 ts_tablet_manager.cc:1397] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 00:48:54.802250 9036 raft_consensus.cc:348] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.802361 9036 raft_consensus.cc:374] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:54.802423 9036 raft_consensus.cc:729] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae19466af3024b9398142b78ec66f846, State: Initialized, Role: FOLLOWER
I20260925 00:48:54.802549 9036 consensus_queue.cc:260] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.802807 9036 ts_tablet_manager.cc:1428] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 00:48:54.891271 9043 raft_consensus.cc:482] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:48:54.891424 9043 raft_consensus.cc:504] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.892503 9043 leader_election.cc:288] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609), 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67:42747)
I20260925 00:48:54.897950 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.893039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:59815 (local address 127.0.249.67:42747)
0925 00:48:54.893172 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.893176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.893233 (+ 57us) server_negotiation.cc:407] Connection header received
0925 00:48:54.893343 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.893346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.893421 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.893501 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:54.893924 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.894713 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.896715 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.897324 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.897529 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.897587 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.897753 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.897826 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:48:54.897997 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.892789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42747 (local address 127.0.249.65:59815)
0925 00:48:54.893149 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.893160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.893239 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.893533 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.893536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.893556 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:54.893802 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.893811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.894850 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.894853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.896568 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.896574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.897525 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.897531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.897629 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.897649 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.897733 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.897805 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":224,"spinlock_wait_cycles":3072,"thread_start_us":82,"threads_started":1}
I20260925 00:48:54.898247 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.892792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:46637 (local address 127.0.249.66:41609)
0925 00:48:54.892965 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.892970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.893077 (+ 107us) server_negotiation.cc:407] Connection header received
0925 00:48:54.893327 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.893331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.893413 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.893508 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:48:54.893924 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.894740 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.896715 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.897385 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.897530 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.897587 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.897940 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.898090 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 00:48:54.898669 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.892675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:41609 (local address 127.0.249.65:46637)
0925 00:48:54.892980 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.892996 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.893157 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.893524 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.893527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.893546 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:48:54.893807 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.893812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.894850 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.894853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.896562 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.896574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.897670 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.897674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.897776 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.897802 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.898401 (+ 599us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.898498 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":259,"thread_start_us":72,"threads_started":1}
I20260925 00:48:54.898792 8962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7225d619f1924268928096ab99da59b4" candidate_uuid: "3a5481de9f0a4087b81b84910652c04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" is_pre_election: true
I20260925 00:48:54.898963 8962 raft_consensus.cc:2454] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5481de9f0a4087b81b84910652c04e in term 0.
I20260925 00:48:54.899137 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7225d619f1924268928096ab99da59b4" candidate_uuid: "3a5481de9f0a4087b81b84910652c04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19466af3024b9398142b78ec66f846" is_pre_election: true
I20260925 00:48:54.899289 8809 raft_consensus.cc:2454] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5481de9f0a4087b81b84910652c04e in term 0.
I20260925 00:48:54.899355 8616 leader_election.cc:302] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5481de9f0a4087b81b84910652c04e, 4d45803d22a141c4abbb3014e5d7ee6e; no voters:
I20260925 00:48:54.899605 9043 raft_consensus.cc:2781] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:48:54.899664 9043 raft_consensus.cc:482] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:48:54.899693 9043 raft_consensus.cc:3037] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.900672 9043 raft_consensus.cc:504] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.901016 9043 leader_election.cc:288] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [CANDIDATE]: Term 1 election: Requested vote from peers ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609), 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67:42747)
I20260925 00:48:54.901372 8962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7225d619f1924268928096ab99da59b4" candidate_uuid: "3a5481de9f0a4087b81b84910652c04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d45803d22a141c4abbb3014e5d7ee6e"
I20260925 00:48:54.901479 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7225d619f1924268928096ab99da59b4" candidate_uuid: "3a5481de9f0a4087b81b84910652c04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19466af3024b9398142b78ec66f846"
I20260925 00:48:54.901551 8962 raft_consensus.cc:3037] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.901592 8809 raft_consensus.cc:3037] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:54.902571 8962 raft_consensus.cc:2454] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5481de9f0a4087b81b84910652c04e in term 1.
I20260925 00:48:54.902596 8809 raft_consensus.cc:2454] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5481de9f0a4087b81b84910652c04e in term 1.
I20260925 00:48:54.902887 8613 leader_election.cc:302] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5481de9f0a4087b81b84910652c04e, ae19466af3024b9398142b78ec66f846; no voters:
I20260925 00:48:54.903040 9043 raft_consensus.cc:2781] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:54.903240 9043 raft_consensus.cc:686] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 1 LEADER]: Becoming Leader. State: Replica: 3a5481de9f0a4087b81b84910652c04e, State: Running, Role: LEADER
I20260925 00:48:54.903484 9043 consensus_queue.cc:237] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } }
I20260925 00:48:54.906880 8525 catalog_manager.cc:5495] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a5481de9f0a4087b81b84910652c04e (127.0.249.65). New cstate: current_term: 1 leader_uuid: "3a5481de9f0a4087b81b84910652c04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5481de9f0a4087b81b84910652c04e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 } health_report { overall_health: UNKNOWN } } }
I20260925 00:48:54.924952 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.921453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.0.249.65:44535)
0925 00:48:54.921720 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.921727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.921739 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:54.921806 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.921808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.921856 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.921956 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:54.922143 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.922668 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.923539 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.923767 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.923836 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.924084 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.924545 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.924638 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.924698 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.924742 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":126,"threads_started":1}
I20260925 00:48:54.934316 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.931618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:51373 (local address 127.0.249.67:42747)
0925 00:48:54.931740 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.931743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.931755 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:54.931860 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.931863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.931905 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.931970 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:54.932154 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.932638 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.933383 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.933648 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.933725 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.933953 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.934015 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.934068 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.934174 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.934217 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:48:54.934334 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.931410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42747 (local address 127.0.249.65:51373)
0925 00:48:54.931697 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.931710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.931780 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.931984 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.931985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.931991 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:54.932076 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.932080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.932718 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.932721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.933298 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.933305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.933738 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.933741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.933807 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.933845 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.934082 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.934087 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.934154 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.934210 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
I20260925 00:48:54.939888 8962 raft_consensus.cc:1261] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Refusing update from remote peer 3a5481de9f0a4087b81b84910652c04e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:54.939961 8809 raft_consensus.cc:1261] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Refusing update from remote peer 3a5481de9f0a4087b81b84910652c04e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:48:54.940483 9043 consensus_queue.cc:1035] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae19466af3024b9398142b78ec66f846" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:54.940697 9130 consensus_queue.cc:1035] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d45803d22a141c4abbb3014e5d7ee6e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:48:54.956416 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.953343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:58779 (local address 127.0.249.65:44535)
0925 00:48:54.953460 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:54.953465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:54.953501 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:48:54.953637 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:54.953639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:54.953674 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:54.953761 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:48:54.953988 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.954659 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.955450 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.955681 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.955784 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.956007 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.956086 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.956161 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:54.956247 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 00:48:54.956292 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 00:48:54.956436 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.953282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:44535 (local address 127.0.249.67:58779)
0925 00:48:54.953442 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.953458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:54.953551 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:54.953771 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:54.953774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:54.953781 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 00:48:54.953883 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.953891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.954762 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.954767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.955340 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:54.955347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:54.955783 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:54.955787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:54.955868 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:54.955900 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 00:48:54.956166 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 00:48:54.956174 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 00:48:54.956237 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:48:54.956315 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 00:48:54.978217 8523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:54.978355 8523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:48:54.980446 8523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 804ccc135e574f848611ef0b4871da13: Sending DeleteTablet for 3 replicas of tablet 7225d619f1924268928096ab99da59b4
I20260925 00:48:54.981045 8784 tablet_service.cc:1514] Processing DeleteTablet for tablet 7225d619f1924268928096ab99da59b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:54 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:34508
I20260925 00:48:54.981042 8942 tablet_service.cc:1514] Processing DeleteTablet for tablet 7225d619f1924268928096ab99da59b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:54 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:51860
I20260925 00:48:54.981057 8662 tablet_service.cc:1514] Processing DeleteTablet for tablet 7225d619f1924268928096ab99da59b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:48:54 UTC) from {username='oryx', principal='oryx/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:50462
I20260925 00:48:54.981233 9105 tablet_replica.cc:330] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: stopping tablet replica
I20260925 00:48:54.981240 9103 tablet_replica.cc:330] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e: stopping tablet replica
I20260925 00:48:54.981335 9105 raft_consensus.cc:2229] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:54.981353 9103 raft_consensus.cc:2229] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:48:54.981473 9105 raft_consensus.cc:2258] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:54.981652 9104 tablet_replica.cc:330] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: stopping tablet replica
I20260925 00:48:54.981660 9103 raft_consensus.cc:2258] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:54.981753 9104 raft_consensus.cc:2229] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:48:54.981874 9104 raft_consensus.cc:2258] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:48:54.982558 9103 ts_tablet_manager.cc:1905] T 7225d619f1924268928096ab99da59b4 P 3a5481de9f0a4087b81b84910652c04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:54.982609 9105 ts_tablet_manager.cc:1905] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:54.982915 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 8597
I20260925 00:48:54.985167 9104 ts_tablet_manager.cc:1905] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:48:54.985257 9105 ts_tablet_manager.cc:1918] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:54.985318 9105 log.cc:1196] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/7225d619f1924268928096ab99da59b4
I20260925 00:48:54.985836 9105 ts_tablet_manager.cc:1939] T 7225d619f1924268928096ab99da59b4 P ae19466af3024b9398142b78ec66f846: Deleting consensus metadata
I20260925 00:48:54.986492 8509 catalog_manager.cc:4841] TS ae19466af3024b9398142b78ec66f846 (127.0.249.66:41609): tablet 7225d619f1924268928096ab99da59b4 (table test-table [id=846de2766b80439494878d3c8b6b453f]) successfully deleted
I20260925 00:48:54.987393 9104 ts_tablet_manager.cc:1918] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 00:48:54.987454 9104 log.cc:1196] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/7225d619f1924268928096ab99da59b4
I20260925 00:48:54.987694 9104 ts_tablet_manager.cc:1939] T 7225d619f1924268928096ab99da59b4 P 4d45803d22a141c4abbb3014e5d7ee6e: Deleting consensus metadata
I20260925 00:48:54.988322 8512 catalog_manager.cc:4841] TS 4d45803d22a141c4abbb3014e5d7ee6e (127.0.249.67:42747): tablet 7225d619f1924268928096ab99da59b4 (table test-table [id=846de2766b80439494878d3c8b6b453f]) successfully deleted
W20260925 00:48:54.990864 8509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:44535 (error 108)
I20260925 00:48:54.991127 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 8737
I20260925 00:48:54.993057 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:54.992566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:44535 (local address 127.0.0.1:50478)
0925 00:48:54.992827 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:54.992923 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:44535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":64,"threads_started":1}
W20260925 00:48:54.993263 8509 catalog_manager.cc:4584] TS 3a5481de9f0a4087b81b84910652c04e (127.0.249.65:44535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7225d619f1924268928096ab99da59b4: Network error: Client connection negotiation failed: client connection to 127.0.249.65:44535: connect: Connection refused (error 111)
I20260925 00:48:54.998293 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 8877
I20260925 00:48:55.005347 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 8493
2026-09-25T00:48:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3444 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:55 dist-test-slave-05m0 krb5kdc[9144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:55 dist-test-slave-05m0 krb5kdc[9144](info): set up 2 sockets
Sep 25 00:48:55 dist-test-slave-05m0 krb5kdc[9144](info): commencing operation
krb5kdc: starting...
W20260925 00:48:56.082160 997 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:56 dist-test-slave-05m0 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:56.200362 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46657
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:56.313130 9157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:56.313457 9157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:56.313530 9157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:56.317444 9157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:56.317504 9157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:56.317526 9157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:56.317546 9157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:56.317564 9157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:56.322853 9157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:56.323799 9157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:56.324761 9157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:56.333151 9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:56.333225 9165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:56.333175 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:56.333534 9157 server_base.cc:1034] running on GCE node
I20260925 00:48:56.334100 9157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:56.335144 9157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:56.336301 9157 hybrid_clock.cc:648] HybridClock initialized: now 1790297336336281 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T00:48:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1341 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:48:56 dist-test-slave-05m0 krb5kdc[9170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:48:56 dist-test-slave-05m0 krb5kdc[9170](info): set up 2 sockets
Sep 25 00:48:56 dist-test-slave-05m0 krb5kdc[9170](info): commencing operation
krb5kdc: starting...
W20260925 00:48:57.418196 997 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:48:57 dist-test-slave-05m0 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:48:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:48:57.536442 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39517
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:39517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:57.643801 9183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:57.644068 9183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:57.644109 9183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:57.647984 9183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:57.648042 9183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:57.648061 9183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:57.648077 9183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:57.648092 9183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:57.653169 9183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:39517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:57.654232 9183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:57.655198 9183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:57.663355 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:57.663373 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:57.663396 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:57.663570 9183 server_base.cc:1034] running on GCE node
I20260925 00:48:57.664146 9183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:57.665220 9183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:57.666399 9183 hybrid_clock.cc:648] HybridClock initialized: now 1790297337666385 us; error 30 us; skew 500 ppm
Sep 25 00:48:57 dist-test-slave-05m0 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:57.674046 9183 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:57.674947 9183 webserver.cc:460] Webserver started at http://127.0.249.126:42737/ using document root <none> and password file <none>
I20260925 00:48:57.675493 9183 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:57.675591 9183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:57.675809 9183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:57.677865 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "333420a9048641ef8e119ced6a4dfaaa"
format_stamp: "Formatted at 2026-09-25 00:48:57 on dist-test-slave-05m0"
server_key: "7739aafeb6dbc9f1fd642094e3178755"
server_key_iv: "2b4397efb29f8a0341e3dc1ec456d096"
server_key_version: "encryptionkey@0"
I20260925 00:48:57.678416 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "333420a9048641ef8e119ced6a4dfaaa"
format_stamp: "Formatted at 2026-09-25 00:48:57 on dist-test-slave-05m0"
server_key: "7739aafeb6dbc9f1fd642094e3178755"
server_key_iv: "2b4397efb29f8a0341e3dc1ec456d096"
server_key_version: "encryptionkey@0"
I20260925 00:48:57.681953 9183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:48:57.684186 9198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:57.685528 9183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:57.685658 9183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "333420a9048641ef8e119ced6a4dfaaa"
format_stamp: "Formatted at 2026-09-25 00:48:57 on dist-test-slave-05m0"
server_key: "7739aafeb6dbc9f1fd642094e3178755"
server_key_iv: "2b4397efb29f8a0341e3dc1ec456d096"
server_key_version: "encryptionkey@0"
I20260925 00:48:57.685772 9183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:57.703878 9183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:57.706808 9183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:57.707013 9183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:57.713840 9183 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:39517
I20260925 00:48:57.713860 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:39517 every 8 connection(s)
I20260925 00:48:57.714763 9183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:57.717414 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:48:57.722524 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 9183
I20260925 00:48:57.722653 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:48:57.722985 997 external_mini_cluster.cc:1428] Setting key 5d1380d49cf1e3dbd74e0abec93dad7f
I20260925 00:48:57.723083 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: Bootstrap starting.
I20260925 00:48:57.725579 9251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: Neither blocks nor log segments found. Creating new log.
I20260925 00:48:57.726301 9251 log.cc:824] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:57.728102 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: No bootstrap required, opened a new log
Sep 25 00:48:57 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:57.732184 9251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } }
I20260925 00:48:57.732435 9251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:48:57.732511 9251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333420a9048641ef8e119ced6a4dfaaa, State: Initialized, Role: FOLLOWER
I20260925 00:48:57.733040 9251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } }
I20260925 00:48:57.733170 9251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:57.733265 9251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:57.733355 9251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:48:57.734259 9251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } }
I20260925 00:48:57.734645 9251 leader_election.cc:302] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 333420a9048641ef8e119ced6a4dfaaa; no voters:
I20260925 00:48:57.735030 9251 leader_election.cc:288] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:48:57.735136 9256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:48:57.735347 9256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 1 LEADER]: Becoming Leader. State: Replica: 333420a9048641ef8e119ced6a4dfaaa, State: Running, Role: LEADER
I20260925 00:48:57.735652 9256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } }
I20260925 00:48:57.736384 9251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:57.736804 9258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333420a9048641ef8e119ced6a4dfaaa. Latest consensus state: current_term: 1 leader_uuid: "333420a9048641ef8e119ced6a4dfaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } } }
I20260925 00:48:57.736930 9258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:57.737174 9257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "333420a9048641ef8e119ced6a4dfaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } } }
I20260925 00:48:57.737287 9257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:57.737303 9260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:57.738472 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:57.724408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.0.249.126:39517)
0925 00:48:57.724845 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:57.724860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:57.724894 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:48:57.725541 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:57.725558 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:57.725928 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:57.726264 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:57.727056 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:57.728004 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:57.728912 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:57.729376 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:57.729597 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:57.732286 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:57.732291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:57.732304 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:57.732332 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:57.734243 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:57.734764 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:57.734773 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:57.734783 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:57.734904 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:57.735242 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:57.735248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:57.735252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:57.735766 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:57.735948 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:57.736380 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 00:48:57.737537 (+ 1157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
I20260925 00:48:57.740413 9260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:57.746552 9260 catalog_manager.cc:1347] Generated new cluster ID: 7cee248c726d43afaaebbcab36f30bb9
I20260925 00:48:57.746616 9260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:57.768723 9260 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:48:57.770093 9260 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:57.775880 9260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: Generated new TSK 0
I20260925 00:48:57.776686 9260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:48:57.807881 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39517
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:57.916041 9277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:57.916280 9277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:57.916344 9277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:57.920388 9277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:57.920444 9277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:57.920532 9277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:57.925612 9277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:57.926569 9277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:57.927527 9277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:57.936616 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:57.936676 9277 server_base.cc:1034] running on GCE node
W20260925 00:48:57.936625 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:57.936633 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:57.937369 9277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:57.938124 9277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:57.939311 9277 hybrid_clock.cc:648] HybridClock initialized: now 1790297337939272 us; error 48 us; skew 500 ppm
Sep 25 00:48:57 dist-test-slave-05m0 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:57.942153 9277 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:48:57.942956 9277 webserver.cc:460] Webserver started at http://127.0.249.65:45225/ using document root <none> and password file <none>
I20260925 00:48:57.943497 9277 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:57.943583 9277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:57.943804 9277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:57.945884 9277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "a617f711c8ec4645bf48c0301f68f574"
format_stamp: "Formatted at 2026-09-25 00:48:57 on dist-test-slave-05m0"
server_key: "0ad1c82d5d57a578cc280fa6dfb7907a"
server_key_iv: "fd461914b89dc1c89b2c3b3f1500915b"
server_key_version: "encryptionkey@0"
I20260925 00:48:57.946429 9277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "a617f711c8ec4645bf48c0301f68f574"
format_stamp: "Formatted at 2026-09-25 00:48:57 on dist-test-slave-05m0"
server_key: "0ad1c82d5d57a578cc280fa6dfb7907a"
server_key_iv: "fd461914b89dc1c89b2c3b3f1500915b"
server_key_version: "encryptionkey@0"
I20260925 00:48:57.949970 9277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:48:57.952207 9292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:57.953366 9277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:57.953527 9277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "a617f711c8ec4645bf48c0301f68f574"
format_stamp: "Formatted at 2026-09-25 00:48:57 on dist-test-slave-05m0"
server_key: "0ad1c82d5d57a578cc280fa6dfb7907a"
server_key_iv: "fd461914b89dc1c89b2c3b3f1500915b"
server_key_version: "encryptionkey@0"
I20260925 00:48:57.953639 9277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:57.968164 9277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:57.971148 9277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:57.971340 9277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:57.971902 9277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:57.972797 9277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:57.972859 9277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:57.972924 9277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:57.972970 9277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:57.981212 9277 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:44589
I20260925 00:48:57.981272 9405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:44589 every 8 connection(s)
I20260925 00:48:57.982122 9277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:57.984123 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 9277
I20260925 00:48:57.984253 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:48:57.984565 997 external_mini_cluster.cc:1428] Setting key 20fbe207777d8f52e602258cf59dba50
Sep 25 00:48:57 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
I20260925 00:48:57.996865 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:57.984017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:39549 (local address 127.0.249.126:39517)
0925 00:48:57.984181 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:57.984188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:57.985084 (+ 896us) server_negotiation.cc:407] Connection header received
0925 00:48:57.986801 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:57.986804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:57.986845 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:57.986922 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:57.988363 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:57.989125 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:57.990458 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:57.990692 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:57.990801 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:57.993196 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:57.993199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:57.993201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:57.993224 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:57.994774 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:57.995325 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:57.995330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:57.995332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:57.995423 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:57.995754 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:57.995759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:57.995761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:57.995983 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:57.996091 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:57.996636 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 00:48:57.996743 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:48:57.998054 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:57.984434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:39549)
0925 00:48:57.984920 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:57.984970 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:57.986576 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:57.987090 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:57.987102 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:57.987638 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:57.988201 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:57.988214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:57.989258 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:57.989262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:57.990329 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:57.990339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:57.990838 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:57.990842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:57.991023 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:57.991661 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:57.991676 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:57.993034 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:57.994942 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:57.994948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:57.994962 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:57.995222 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:57.995523 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:57.995526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:57.995528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:57.995660 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:57.996089 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:57.996096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:57.996358 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 00:48:57.996593 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 00:48:57.996963 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":149,"threads_started":1}
I20260925 00:48:57.999307 9406 heartbeater.cc:347] Connected to a master server at 127.0.249.126:39517
I20260925 00:48:57.999577 9406 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:58.000116 9406 heartbeater.cc:511] Master 127.0.249.126:39517 requested a full tablet report, sending...
I20260925 00:48:58.001708 9215 ts_manager.cc:194] Registered new tserver with Master: a617f711c8ec4645bf48c0301f68f574 (127.0.249.65:44589)
I20260925 00:48:58.002863 9215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:39549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:48:58.010637 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39517
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:58.122339 9411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:58.122643 9411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:58.122726 9411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:58.126925 9411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:58.127017 9411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:58.127115 9411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:58.132424 9411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:58.133657 9411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:58.134693 9411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:58.144390 9419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:58.144356 9417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:58.144816 9411 server_base.cc:1034] running on GCE node
W20260925 00:48:58.144359 9416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:58.145223 9411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:58.145886 9411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:58.147060 9411 hybrid_clock.cc:648] HybridClock initialized: now 1790297338147046 us; error 32 us; skew 500 ppm
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:58.149770 9411 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:48:58.150574 9411 webserver.cc:460] Webserver started at http://127.0.249.66:36009/ using document root <none> and password file <none>
I20260925 00:48:58.151120 9411 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:58.151206 9411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:58.151418 9411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:58.153321 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "a8f12330369841719701dd95b17d7389"
format_stamp: "Formatted at 2026-09-25 00:48:58 on dist-test-slave-05m0"
server_key: "7615cbb6302a5d8683b9206de8e36c80"
server_key_iv: "808d1b6ceaef31a8a7d15a9d6484b5b1"
server_key_version: "encryptionkey@0"
I20260925 00:48:58.153923 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "a8f12330369841719701dd95b17d7389"
format_stamp: "Formatted at 2026-09-25 00:48:58 on dist-test-slave-05m0"
server_key: "7615cbb6302a5d8683b9206de8e36c80"
server_key_iv: "808d1b6ceaef31a8a7d15a9d6484b5b1"
server_key_version: "encryptionkey@0"
I20260925 00:48:58.157465 9411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:58.159741 9426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.160929 9411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:58.161051 9411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "a8f12330369841719701dd95b17d7389"
format_stamp: "Formatted at 2026-09-25 00:48:58 on dist-test-slave-05m0"
server_key: "7615cbb6302a5d8683b9206de8e36c80"
server_key_iv: "808d1b6ceaef31a8a7d15a9d6484b5b1"
server_key_version: "encryptionkey@0"
I20260925 00:48:58.161150 9411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:58.172335 9411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:58.175485 9411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:58.175686 9411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:58.176229 9411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:58.176994 9411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:58.177060 9411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.177127 9411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:58.177173 9411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.185621 9411 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:35451
I20260925 00:48:58.185667 9539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:35451 every 8 connection(s)
I20260925 00:48:58.186470 9411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:58.196627 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 9411
I20260925 00:48:58.196770 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:48:58.197081 997 external_mini_cluster.cc:1428] Setting key 5c3fe19c1a0077aca9930a47c2c946aa
I20260925 00:48:58.200039 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:58.188147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:57919 (local address 127.0.249.126:39517)
0925 00:48:58.188301 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:58.188306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:58.189101 (+ 795us) server_negotiation.cc:407] Connection header received
0925 00:48:58.190625 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:58.190627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:58.190672 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:58.190745 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:58.191830 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.192339 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.193458 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.193679 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.193757 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.196116 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:58.196120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:58.196123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:58.196147 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:58.197852 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:58.198423 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:58.198430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:58.198435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:58.198532 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:58.198866 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:58.198869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:58.198871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:58.199105 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:58.199206 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:58.199659 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 00:48:58.199802 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:48:58.200657 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:58.188543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:57919)
0925 00:48:58.188962 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:58.189003 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:58.190435 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:58.190898 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:58.190904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:58.191230 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:58.191703 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.191716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.192445 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.192448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.193304 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.193313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.193848 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.193852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.194002 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.194534 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:58.194549 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:58.195960 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:58.197965 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:58.197971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:58.197984 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:58.198311 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:58.198627 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:58.198630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:58.198632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:58.198771 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:58.199210 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:58.199215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:58.199423 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 00:48:58.199582 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 00:48:58.199815 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20260925 00:48:58.201828 9540 heartbeater.cc:347] Connected to a master server at 127.0.249.126:39517
I20260925 00:48:58.202046 9540 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:58.202487 9540 heartbeater.cc:511] Master 127.0.249.126:39517 requested a full tablet report, sending...
I20260925 00:48:58.203459 9215 ts_manager.cc:194] Registered new tserver with Master: a8f12330369841719701dd95b17d7389 (127.0.249.66:35451)
I20260925 00:48:58.204021 9215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:57919
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:48:58.224046 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39517
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:58.335049 9545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:58.335289 9545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:58.335330 9545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:58.339514 9545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:58.339570 9545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:58.339627 9545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:58.344892 9545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:58.345896 9545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:58.346801 9545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:58.356463 9553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:58.356463 9551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:58.356506 9550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:58.356880 9545 server_base.cc:1034] running on GCE node
I20260925 00:48:58.357266 9545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:58.358016 9545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:58.359181 9545 hybrid_clock.cc:648] HybridClock initialized: now 1790297338359151 us; error 48 us; skew 500 ppm
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:58.361871 9545 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:48:58.362668 9545 webserver.cc:460] Webserver started at http://127.0.249.67:42171/ using document root <none> and password file <none>
I20260925 00:48:58.363204 9545 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:58.363286 9545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:58.363503 9545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:48:58.365613 9545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "47c0ad2343f043899f61bf02c21df905"
format_stamp: "Formatted at 2026-09-25 00:48:58 on dist-test-slave-05m0"
server_key: "b9cd69109612543f8591db99054da2b8"
server_key_iv: "5bfe6cb1b5bdcc280099aba7115964b3"
server_key_version: "encryptionkey@0"
I20260925 00:48:58.366144 9545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "47c0ad2343f043899f61bf02c21df905"
format_stamp: "Formatted at 2026-09-25 00:48:58 on dist-test-slave-05m0"
server_key: "b9cd69109612543f8591db99054da2b8"
server_key_iv: "5bfe6cb1b5bdcc280099aba7115964b3"
server_key_version: "encryptionkey@0"
I20260925 00:48:58.369597 9545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:58.371830 9560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.373008 9545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:58.373128 9545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "47c0ad2343f043899f61bf02c21df905"
format_stamp: "Formatted at 2026-09-25 00:48:58 on dist-test-slave-05m0"
server_key: "b9cd69109612543f8591db99054da2b8"
server_key_iv: "5bfe6cb1b5bdcc280099aba7115964b3"
server_key_version: "encryptionkey@0"
I20260925 00:48:58.373227 9545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:58.390725 9545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:58.393913 9545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:58.394109 9545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:58.394672 9545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:58.395463 9545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:58.395527 9545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.395592 9545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:58.395637 9545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.404242 9545 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:34305
I20260925 00:48:58.404270 9673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:34305 every 8 connection(s)
I20260925 00:48:58.405117 9545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:58.410198 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 9545
I20260925 00:48:58.410300 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:48:58.410625 997 external_mini_cluster.cc:1428] Setting key 93e7433abc387e15afbbf1b32f678892
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:48:58.418401 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:58.407023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:49323 (local address 127.0.249.126:39517)
0925 00:48:58.407160 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:58.407163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:58.407989 (+ 826us) server_negotiation.cc:407] Connection header received
0925 00:48:58.409320 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:58.409323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:58.409381 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:58.409480 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:58.410686 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.411264 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.412599 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.412859 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.412932 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.415244 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:58.415247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:58.415249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:58.415274 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:58.416703 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:58.417150 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:58.417152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:58.417153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:58.417223 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:58.417508 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:58.417510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:58.417511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:58.417680 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:58.417756 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:58.418160 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 00:48:58.418273 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:48:58.419188 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:58.407394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:49323)
0925 00:48:58.407810 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:58.407850 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:58.409116 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:58.409657 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:58.409664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:58.409996 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:58.410547 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.410561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.411428 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.411435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.412464 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.412478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.412988 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.412990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.413127 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.413690 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:58.413705 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:58.415089 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:48:58.416798 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:58.416802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:58.416813 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:58.417071 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:58.417295 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:48:58.417297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:48:58.417299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:48:58.417439 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:48:58.417762 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:48:58.417766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:48:58.417970 (+ 204us) client_negotiation.cc:769] Sending connection context
0925 00:48:58.418117 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 00:48:58.418344 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":111,"threads_started":1}
I20260925 00:48:58.420295 9674 heartbeater.cc:347] Connected to a master server at 127.0.249.126:39517
I20260925 00:48:58.420544 9674 heartbeater.cc:464] Registering TS with master...
I20260925 00:48:58.421012 9674 heartbeater.cc:511] Master 127.0.249.126:39517 requested a full tablet report, sending...
I20260925 00:48:58.422024 9215 ts_manager.cc:194] Registered new tserver with Master: 47c0ad2343f043899f61bf02c21df905 (127.0.249.67:34305)
I20260925 00:48:58.422593 9215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:49323
I20260925 00:48:58.424448 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab.
Entry for principal oryx/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab.
WARNING: no policy specified for oryx/127.0.249.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab.
Entry for principal oryx/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab.
WARNING: no policy specified for oryx/127.0.249.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab.
Entry for principal oryx/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab.
I20260925 00:48:58.492834 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 9277
I20260925 00:48:58.498533 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 9411
I20260925 00:48:58.503702 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 9545
I20260925 00:48:58.508369 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 9183
I20260925 00:48:58.514417 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39517
--webserver_interface=127.0.249.126
--webserver_port=42737
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:39517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:48:58.628466 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:58.628772 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:58.628860 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:58.632850 9683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:48:58.632974 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:58.633018 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:58.633057 9683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:48:58.633098 9683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:48:58.638352 9683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:39517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=42737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:58.639470 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:58.640498 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:58.648967 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:58.649005 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:58.649181 9683 server_base.cc:1034] running on GCE node
W20260925 00:48:58.649060 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:58.649950 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:58.651016 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:58.652196 9683 hybrid_clock.cc:648] HybridClock initialized: now 1790297338652187 us; error 34 us; skew 500 ppm
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:58.654937 9683 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:48:58.655798 9683 webserver.cc:460] Webserver started at http://127.0.249.126:42737/ using document root <none> and password file <none>
I20260925 00:48:58.656348 9683 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:58.656456 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:58.660339 9683 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:58.662510 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.663781 9683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:58.663934 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "333420a9048641ef8e119ced6a4dfaaa"
format_stamp: "Formatted at 2026-09-25 00:48:57 on dist-test-slave-05m0"
server_key: "7739aafeb6dbc9f1fd642094e3178755"
server_key_iv: "2b4397efb29f8a0341e3dc1ec456d096"
server_key_version: "encryptionkey@0"
I20260925 00:48:58.664484 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:58.678716 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:58.681775 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:58.681977 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:58.689358 9683 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:39517
I20260925 00:48:58.689378 9750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:39517 every 8 connection(s)
I20260925 00:48:58.690512 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:48:58.694785 9751 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:48:58.696475 9751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: Bootstrap starting.
I20260925 00:48:58.700495 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 9683
I20260925 00:48:58.701025 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:44589
--local_ip_for_outbound_sockets=127.0.249.65
--tserver_master_addrs=127.0.249.126:39517
--webserver_port=45225
--webserver_interface=127.0.249.65
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.249.65
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:48:58.715020 9751 log.cc:824] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: Log is configured to *not* fsync() on all Append() calls
I20260925 00:48:58.722074 9751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:48:58.722496 9751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: Bootstrap complete.
I20260925 00:48:58.726898 9751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } }
I20260925 00:48:58.727308 9751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333420a9048641ef8e119ced6a4dfaaa, State: Initialized, Role: FOLLOWER
I20260925 00:48:58.727820 9751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } }
I20260925 00:48:58.727941 9751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:48:58.728029 9751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:48:58.728145 9751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:48:58.730283 9751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } }
I20260925 00:48:58.730656 9751 leader_election.cc:302] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 333420a9048641ef8e119ced6a4dfaaa; no voters:
I20260925 00:48:58.731036 9751 leader_election.cc:288] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 00:48:58.731135 9755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:48:58.731362 9755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [term 2 LEADER]: Becoming Leader. State: Replica: 333420a9048641ef8e119ced6a4dfaaa, State: Running, Role: LEADER
I20260925 00:48:58.731675 9755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } }
I20260925 00:48:58.732331 9751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:48:58.733284 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333420a9048641ef8e119ced6a4dfaaa. Latest consensus state: current_term: 2 leader_uuid: "333420a9048641ef8e119ced6a4dfaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } } }
I20260925 00:48:58.733290 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "333420a9048641ef8e119ced6a4dfaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333420a9048641ef8e119ced6a4dfaaa" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39517 } } }
I20260925 00:48:58.733479 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:58.733490 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa [sys.catalog]: This master's current role is: LEADER
I20260925 00:48:58.734647 9763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:48:58.737872 9763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:48:58.738539 9763 catalog_manager.cc:1259] Loaded cluster ID: 7cee248c726d43afaaebbcab36f30bb9
I20260925 00:48:58.738620 9763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:48:58.740286 9763 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:48:58.740859 9763 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 333420a9048641ef8e119ced6a4dfaaa: Loaded TSK: 0
I20260925 00:48:58.741853 9763 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:48:58.832032 9753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:58.832324 9753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:58.832414 9753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:58.836964 9753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:58.837036 9753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:58.837121 9753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:48:58.842293 9753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.65.keytab
--principal=oryx/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:44589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=45225
--tserver_master_addrs=127.0.249.126:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:58.843295 9753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:58.844358 9753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:58.853544 9778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:58.853555 9780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:58.853623 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:58.853922 9753 server_base.cc:1034] running on GCE node
I20260925 00:48:58.854282 9753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:58.854873 9753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:58.856062 9753 hybrid_clock.cc:648] HybridClock initialized: now 1790297338856035 us; error 42 us; skew 500 ppm
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297338, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:58.858788 9753 init.cc:377] Logged in from keytab as oryx/127.0.249.65@KRBTEST.COM (short username oryx)
I20260925 00:48:58.859637 9753 webserver.cc:460] Webserver started at http://127.0.249.65:45225/ using document root <none> and password file <none>
I20260925 00:48:58.860175 9753 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:58.860245 9753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:58.864171 9753 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260925 00:48:58.866456 9787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.867993 9753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:48:58.868131 9753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "a617f711c8ec4645bf48c0301f68f574"
format_stamp: "Formatted at 2026-09-25 00:48:57 on dist-test-slave-05m0"
server_key: "0ad1c82d5d57a578cc280fa6dfb7907a"
server_key_iv: "fd461914b89dc1c89b2c3b3f1500915b"
server_key_version: "encryptionkey@0"
I20260925 00:48:58.868759 9753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:58.908080 9753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:58.911613 9753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:58.911839 9753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:58.912465 9753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:58.913335 9753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:58.913439 9753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.913530 9753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:58.913579 9753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:58.923110 9753 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:44589
I20260925 00:48:58.923161 9900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:44589 every 8 connection(s)
I20260925 00:48:58.924201 9753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:48:58.928143 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 9753
I20260925 00:48:58.928634 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:35451
--local_ip_for_outbound_sockets=127.0.249.66
--tserver_master_addrs=127.0.249.126:39517
--webserver_port=36009
--webserver_interface=127.0.249.66
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.249.66
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:58.940605 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:58.926553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:60579)
0925 00:48:58.927062 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:58.927104 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:58.928637 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:58.930173 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:58.930183 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:58.930719 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:58.931415 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.931433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.933536 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.933543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.935122 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.935138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.935920 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.935925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.936157 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.936868 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:58.936895 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:58.939653 (+ 2758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
W20260925 00:48:58.940596 9903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:58.926445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:60579 (local address 127.0.249.126:39517)
0925 00:48:58.926979 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:58.926998 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:58.927210 (+ 212us) server_negotiation.cc:407] Connection header received
0925 00:48:58.929213 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:58.929231 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:58.929659 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:58.930041 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:58.931623 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.933277 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.935272 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.935743 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.935892 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.939879 (+ 3987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:60579: BlockingRecv error: recv got EOF from 127.0.249.65:60579 (error 108)
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":144,"threads_started":1}
W20260925 00:48:58.941270 9901 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:39517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.249.126:39517: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:58.947016 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:58.941722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:51465)
0925 00:48:58.941961 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:58.941973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:58.942042 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:58.942267 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:58.942269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:58.942397 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:58.942481 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.942485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.943482 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.943484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.944023 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.944030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.944595 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.944598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.944665 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.945150 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:58.945161 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:58.946928 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181}
W20260925 00:48:58.947227 9903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:58.941829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:51465 (local address 127.0.249.126:39517)
0925 00:48:58.941952 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:58.941957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:58.942021 (+ 64us) server_negotiation.cc:407] Connection header received
0925 00:48:58.942120 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:58.942123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:58.942171 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:58.942248 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:58.942577 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.943376 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.944123 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.944479 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.944586 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.947104 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:51465: BlockingRecv error: recv got EOF from 127.0.249.65:51465 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:58 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:58.953635 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:58.947509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:53799)
0925 00:48:58.947852 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:58.947868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:58.947972 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:58.948194 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:58.948198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:58.948416 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:58.948534 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.948542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.949199 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.949203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.950289 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:58.950303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.950802 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.950806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:58.950967 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.951497 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:58.951512 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:58.953487 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
W20260925 00:48:58.953738 9903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:58.947595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:53799 (local address 127.0.249.126:39517)
0925 00:48:58.947718 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:58.947723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:58.947944 (+ 221us) server_negotiation.cc:407] Connection header received
0925 00:48:58.948054 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:58.948056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:58.948092 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:58.948156 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:58.948637 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.949094 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.950481 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:58.950687 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:58.950767 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:58.953612 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:53799: BlockingRecv error: recv got EOF from 127.0.249.65:53799 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 00:48:58.953912 9901 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:48:59.044800 9905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:59.045037 9905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:59.045077 9905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:59.049175 9905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:59.049228 9905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:59.049300 9905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:48:59.054459 9905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.66.keytab
--principal=oryx/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=36009
--tserver_master_addrs=127.0.249.126:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:59.055408 9905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:59.056375 9905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:59.065680 9914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:59.065649 9911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:59.065719 9905 server_base.cc:1034] running on GCE node
W20260925 00:48:59.065670 9912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:59.066362 9905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:59.066893 9905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:59.068054 9905 hybrid_clock.cc:648] HybridClock initialized: now 1790297339068023 us; error 40 us; skew 500 ppm
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297339, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:59.070757 9905 init.cc:377] Logged in from keytab as oryx/127.0.249.66@KRBTEST.COM (short username oryx)
I20260925 00:48:59.071599 9905 webserver.cc:460] Webserver started at http://127.0.249.66:36009/ using document root <none> and password file <none>
I20260925 00:48:59.072162 9905 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:59.072261 9905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:59.076040 9905 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:59.078040 9921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:59.079169 9905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:59.079290 9905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "a8f12330369841719701dd95b17d7389"
format_stamp: "Formatted at 2026-09-25 00:48:58 on dist-test-slave-05m0"
server_key: "7615cbb6302a5d8683b9206de8e36c80"
server_key_iv: "808d1b6ceaef31a8a7d15a9d6484b5b1"
server_key_version: "encryptionkey@0"
I20260925 00:48:59.079846 9905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:59.093184 9905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:59.095947 9905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:59.096138 9905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:59.096701 9905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:59.097522 9905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:59.097582 9905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:59.097645 9905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:59.097688 9905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:59.106223 9905 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:35451
I20260925 00:48:59.106246 10034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:35451 every 8 connection(s)
I20260925 00:48:59.107192 9905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:48:59.115602 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 9905
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:59.116026 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:34305
--local_ip_for_outbound_sockets=127.0.249.67
--tserver_master_addrs=127.0.249.126:39517
--webserver_port=42171
--webserver_interface=127.0.249.67
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.249.67
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 00:48:59.118019 9903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:59.108854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:57565 (local address 127.0.249.126:39517)
0925 00:48:59.108981 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:59.108984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:59.109851 (+ 867us) server_negotiation.cc:407] Connection header received
0925 00:48:59.111186 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:59.111189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:59.111251 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:59.111355 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:59.112446 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.112961 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.114120 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.114331 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.114436 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.117857 (+ 3421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:57565: BlockingRecv error: recv got EOF from 127.0.249.66:57565 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:48:59.118444 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:59.109237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:57565)
0925 00:48:59.109718 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:59.109757 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:59.110989 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:59.111547 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:59.111553 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:59.111877 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:59.112310 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.112323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.113088 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.113092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.113997 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.114006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.114467 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.114471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.114637 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.115197 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:59.115212 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:59.117684 (+ 2472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":186,"threads_started":1}
W20260925 00:48:59.118914 10035 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:39517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.249.126:39517: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:59.124401 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:59.119333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:33171)
0925 00:48:59.119432 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:59.119447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:59.119521 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:59.119755 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:59.119758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:59.119908 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:59.119997 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.120001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.120685 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.120691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.121253 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.121261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.121726 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.121730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.121816 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.122380 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:59.122392 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:59.124268 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 00:48:59.124532 9903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:59.119423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:33171 (local address 127.0.249.126:39517)
0925 00:48:59.119540 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:59.119546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:59.119560 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:59.119617 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:59.119620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:59.119664 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:59.119742 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:59.120102 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.120585 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.121368 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.121621 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.121728 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.124415 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:33171: BlockingRecv error: recv got EOF from 127.0.249.66:33171 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:59.129819 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:59.125162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:50101)
0925 00:48:59.125250 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:59.125265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:59.125346 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:59.125623 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:59.125626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:59.125767 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:59.125830 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.125834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.126519 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.126525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.127052 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.127059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.127474 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.127477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.127554 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.127997 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:59.128008 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:59.129690 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 00:48:59.130019 10035 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:48:59.130141 9903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:59.125250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:50101 (local address 127.0.249.126:39517)
0925 00:48:59.125382 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:59.125447 (+ 65us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:59.125461 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:59.125495 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:59.125497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:59.125530 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:59.125609 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:59.125940 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.126419 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.127164 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.127369 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.127477 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.129792 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:50101: BlockingRecv error: recv got EOF from 127.0.249.66:50101 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 00:48:59.226292 10038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:48:59.226564 10038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:48:59.226635 10038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:48:59.230695 10038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:48:59.230752 10038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:48:59.230839 10038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:48:59.236557 10038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/krb5kdc/oryx_127.0.249.67.keytab
--principal=oryx/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:34305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=42171
--tserver_master_addrs=127.0.249.126:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:48:59.237636 10038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:48:59.238765 10038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:48:59.247991 10046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:48:59.248027 10044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:59.248134 10038 server_base.cc:1034] running on GCE node
W20260925 00:48:59.248027 10043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:48:59.248603 10038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:48:59.249240 10038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:48:59.250408 10038 hybrid_clock.cc:648] HybridClock initialized: now 1790297339250375 us; error 45 us; skew 500 ppm
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297339, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:48:59.253029 10038 init.cc:377] Logged in from keytab as oryx/127.0.249.67@KRBTEST.COM (short username oryx)
I20260925 00:48:59.253913 10038 webserver.cc:460] Webserver started at http://127.0.249.67:42171/ using document root <none> and password file <none>
I20260925 00:48:59.254407 10038 fs_manager.cc:359] Metadata directory not provided
I20260925 00:48:59.254474 10038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:48:59.258172 10038 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:48:59.260164 10053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:59.261479 10038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:48:59.261579 10038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "47c0ad2343f043899f61bf02c21df905"
format_stamp: "Formatted at 2026-09-25 00:48:58 on dist-test-slave-05m0"
server_key: "b9cd69109612543f8591db99054da2b8"
server_key_iv: "5bfe6cb1b5bdcc280099aba7115964b3"
server_key_version: "encryptionkey@0"
I20260925 00:48:59.262238 10038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:48:59.276830 10038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:48:59.279950 10038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:48:59.280097 10038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:48:59.280589 10038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:48:59.281311 10038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:48:59.281358 10038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:59.281438 10038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:48:59.281472 10038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:48:59.290277 10038 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:34305
I20260925 00:48:59.290306 10166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:34305 every 8 connection(s)
I20260925 00:48:59.291220 10038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:48:59.292598 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 10038
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:59.302610 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:59.293949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.0.249.126:39517)
0925 00:48:59.294155 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:59.294159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:59.294169 (+ 10us) server_negotiation.cc:407] Connection header received
0925 00:48:59.294208 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:59.294216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:59.294255 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:59.294355 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:59.295104 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.295775 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.296635 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.296884 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.296990 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.298457 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:48:59.298468 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:48:59.298483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:48:59.298509 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:48:59.300321 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:59.300802 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:59.300807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:59.300812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:59.300941 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:48:59.301241 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:48:59.301245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:48:59.301246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:48:59.301761 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:48:59.301985 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 00:48:59.302181 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 00:48:59.302445 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
W20260925 00:48:59.302620 9903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:59.293033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:44927 (local address 127.0.249.126:39517)
0925 00:48:59.293165 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:59.293170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:59.294124 (+ 954us) server_negotiation.cc:407] Connection header received
0925 00:48:59.295482 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:59.295485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:59.295531 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:59.295612 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:59.296710 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.297204 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.298536 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.298765 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.298867 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.302535 (+ 3668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:44927: BlockingRecv error: recv got EOF from 127.0.249.67:44927 (error 108)
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":44}
I20260925 00:48:59.303256 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:59.293538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:44927)
0925 00:48:59.293982 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:59.294041 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:59.295299 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:59.295767 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:59.295774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:59.296114 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:59.296568 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.296581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.297497 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.297502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.298401 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.298410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.298905 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.298909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.299078 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.299525 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:59.299540 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:59.302288 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":131,"threads_started":1}
W20260925 00:48:59.304020 10167 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:39517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.249.126:39517: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:59.309281 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:59.304481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:43105)
0925 00:48:59.304600 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:59.304612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:59.304671 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:59.304911 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:59.304914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:59.305091 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:59.305197 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.305204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.305968 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.305971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.306543 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.306552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.306931 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.306934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.306998 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.307471 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:59.307489 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:59.309148 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 00:48:59.309365 10171 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:59.304621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:43105 (local address 127.0.249.126:39517)
0925 00:48:59.304721 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:59.304725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:59.304737 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:48:59.304770 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:59.304773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:59.304815 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:59.304901 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:59.305309 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.305876 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.306638 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.306846 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.306917 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.309258 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:43105: BlockingRecv error: recv got EOF from 127.0.249.67:43105 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:59.314193 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:59.309779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:47325)
0925 00:48:59.309895 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:59.309907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:59.309990 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:59.310214 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:59.310216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:59.310372 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:59.310435 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.310439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.311115 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.311120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.311624 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.311631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.312036 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.312040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.312116 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.312496 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:59.312516 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:59.314065 (+ 1549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 00:48:59.314271 10171 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:59.309866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:47325 (local address 127.0.249.126:39517)
0925 00:48:59.310044 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:59.310050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:59.310064 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:48:59.310098 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:59.310100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:59.310128 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:59.310201 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:59.310555 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.311017 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.311740 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.311939 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.312046 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.314162 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:47325: BlockingRecv error: recv got EOF from 127.0.249.67:47325 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 00:48:59.314440 10167 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:48:59 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:48:59.959965 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:48:59.954692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:48815)
0925 00:48:59.955036 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 00:48:59.955050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:48:59.955139 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:48:59.955424 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:48:59.955427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:48:59.955641 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:48:59.955738 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.955744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.956484 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.956487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.957073 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:48:59.957081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.957518 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.957520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:48:59.957590 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.958060 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:48:59.958071 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:48:59.959824 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":79,"threads_started":1}
W20260925 00:48:59.960203 10173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:48:59.954745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:48815 (local address 127.0.249.126:39517)
0925 00:48:59.955052 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 00:48:59.955057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:48:59.955094 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:48:59.955245 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:48:59.955248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:48:59.955300 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:48:59.955402 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:48:59.955839 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.956389 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.957171 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:48:59.957423 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:48:59.957498 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:48:59.960026 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:48815: BlockingRecv error: recv got EOF from 127.0.249.65:48815 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":102,"threads_started":1}
Sep 25 00:49:00 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:00 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:00.135746 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:00.130593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:41047)
0925 00:49:00.130826 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:00.130841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:00.130949 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:00.131154 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:00.131156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:00.131324 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:00.131421 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:00.131426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.132176 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.132182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:00.132748 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:00.132757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.133159 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.133161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:00.133249 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:00.133767 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:00.133778 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:00.135624 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
W20260925 00:49:00.135902 10173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:00.130635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:41047 (local address 127.0.249.126:39517)
0925 00:49:00.130793 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:00.130796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:00.130917 (+ 121us) server_negotiation.cc:407] Connection header received
0925 00:49:00.131013 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:00.131015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:00.131057 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:00.131127 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:00.131513 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.132049 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.132852 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.133068 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.133145 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:00.135751 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:41047: BlockingRecv error: recv got EOF from 127.0.249.66:41047 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 00:49:00 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:00 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:00.320390 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:00.315050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:56491)
0925 00:49:00.315329 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:00.315346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:00.315454 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:00.315700 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:00.315702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:00.315877 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:00.315976 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:00.315981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.316702 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.316705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:00.317308 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:00.317317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.317933 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.317936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:00.318005 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:00.318471 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:00.318482 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:00.320256 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
W20260925 00:49:00.320518 10173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:00.315094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:56491 (local address 127.0.249.126:39517)
0925 00:49:00.315244 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:00.315249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:00.315443 (+ 194us) server_negotiation.cc:407] Connection header received
0925 00:49:00.315552 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:00.315554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:00.315603 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:00.315664 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:00.316069 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.316604 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.317568 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.317808 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.317891 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:00.320385 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:56491: BlockingRecv error: recv got EOF from 127.0.249.67:56491 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 00:49:00 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:00 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:00.966153 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:00.960769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:47277)
0925 00:49:00.961064 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:00.961087 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:00.961184 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:00.961442 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:00.961444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:00.961617 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:00.961711 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:00.961716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.962458 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.962460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:00.962996 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:00.963003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.963508 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.963514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:00.963604 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:00.964206 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:00.964217 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:00.966040 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
W20260925 00:49:00.966333 10177 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:00.960923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:47277 (local address 127.0.249.126:39517)
0925 00:49:00.961167 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:00.961171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:00.961186 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:49:00.961265 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:00.961267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:00.961309 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:00.961428 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:00.961806 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.962362 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.963133 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:00.963385 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:00.963492 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:00.966158 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:47277: BlockingRecv error: recv got EOF from 127.0.249.65:47277 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
Sep 25 00:49:01 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:01 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:01.141853 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:01.136502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:38415)
0925 00:49:01.136783 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:01.136796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:01.136918 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:01.137189 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:01.137192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:01.137430 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:01.137539 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:01.137545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.138293 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.138298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:01.138863 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:01.138871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.139307 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.139311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:01.139397 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:01.139892 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:01.139904 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:01.141718 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
W20260925 00:49:01.141985 10177 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:01.136545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:38415 (local address 127.0.249.126:39517)
0925 00:49:01.136712 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:01.136717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:01.136870 (+ 153us) server_negotiation.cc:407] Connection header received
0925 00:49:01.137039 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:01.137041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:01.137085 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:01.137169 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:01.137665 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.138187 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.138978 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.139201 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.139307 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:01.141851 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:38415: BlockingRecv error: recv got EOF from 127.0.249.66:38415 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 00:49:01 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:01 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:01.326953 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:01.321227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:43705)
0925 00:49:01.321535 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:01.321553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:01.321663 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:01.321932 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:01.321935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:01.322132 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:01.322224 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:01.322229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.322973 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.322979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:01.323541 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:01.323549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.324014 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.324018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:01.324099 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:01.324653 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:01.324665 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:01.326768 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
W20260925 00:49:01.326989 10177 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:01.321472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:43705 (local address 127.0.249.126:39517)
0925 00:49:01.321625 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:01.321630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:01.321643 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:49:01.321771 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:01.321775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:01.321821 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:01.321910 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:01.322342 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.322869 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.323659 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.323905 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.324018 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:01.326888 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:43705: BlockingRecv error: recv got EOF from 127.0.249.67:43705 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 00:49:01 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:01 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:01.972776 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:01.966967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:42687)
0925 00:49:01.967225 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:01.967238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:01.967342 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:01.967768 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:01.967771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:01.967990 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:01.968090 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:01.968096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.968961 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.968966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:01.969636 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:01.969645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.970090 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.970094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:01.970189 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:01.970738 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:01.970750 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:01.972581 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
W20260925 00:49:01.972851 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:01.967035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:42687 (local address 127.0.249.126:39517)
0925 00:49:01.967488 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:01.967493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:01.967510 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:49:01.967574 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:01.967577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:01.967629 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:01.967734 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:01.968207 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.968860 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.969746 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:01.969977 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:01.970090 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:01.972717 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:42687: BlockingRecv error: recv got EOF from 127.0.249.65:42687 (error 108)
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
Sep 25 00:49:02 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:02 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:02.147663 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:02.142603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:44293)
0925 00:49:02.142873 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:02.142886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:02.142981 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:02.143185 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:02.143187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:02.143360 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:02.143476 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:02.143482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.144170 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.144173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:02.144750 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:02.144758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.145148 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.145151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:02.145208 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:02.145716 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:02.145728 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:02.147498 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
W20260925 00:49:02.147722 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:02.142673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:44293 (local address 127.0.249.126:39517)
0925 00:49:02.142806 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:02.142809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:02.142937 (+ 128us) server_negotiation.cc:407] Connection header received
0925 00:49:02.143045 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:02.143048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:02.143087 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:02.143151 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:02.143571 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.144079 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.144846 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.145056 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.145127 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:02.147619 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:44293: BlockingRecv error: recv got EOF from 127.0.249.66:44293 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 00:49:02 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:02 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:02.333010 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:02.327776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:59833)
0925 00:49:02.328018 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:02.328038 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:02.328156 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:02.328449 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:02.328452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:02.328631 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:02.328729 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:02.328735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.329516 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.329520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:02.330079 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:02.330087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.330579 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.330584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:02.330678 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:02.331138 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:02.331148 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:02.332898 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
W20260925 00:49:02.333272 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:02.327826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:59833 (local address 127.0.249.126:39517)
0925 00:49:02.328015 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:02.328019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:02.328110 (+ 91us) server_negotiation.cc:407] Connection header received
0925 00:49:02.328245 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:02.328247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:02.328322 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:02.328405 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:02.328832 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.329374 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.330196 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.330443 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.330554 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:02.333041 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:59833: BlockingRecv error: recv got EOF from 127.0.249.67:59833 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 00:49:02 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:02 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:02.989094 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:02.973556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:39129)
0925 00:49:02.973815 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:02.973829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:02.973917 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:02.974168 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:02.974170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:02.974345 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:02.974435 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:02.974440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.975171 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.975173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:02.975748 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:02.975756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.976194 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.976196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:02.976255 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:02.986899 (+ 10644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:02.986913 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:02.988849 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":68,"threads_started":1}
W20260925 00:49:02.989336 10185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:02.973568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:39129 (local address 127.0.249.126:39517)
0925 00:49:02.973849 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:02.973854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:02.973869 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:49:02.974006 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:02.974008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:02.974052 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:02.974131 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:02.974533 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.975076 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.975856 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:02.976098 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:02.976178 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:02.989021 (+ 12843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:39129: BlockingRecv error: recv got EOF from 127.0.249.65:39129 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
Sep 25 00:49:03 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:03 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:03.154255 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:03.148539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:36153)
0925 00:49:03.148851 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:03.148865 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:03.148970 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:03.149198 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:03.149200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:03.149434 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:03.149540 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:03.149546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.150265 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.150268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:03.151145 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:03.151156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.151668 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.151671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:03.151735 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:03.152250 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:03.152263 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:03.154137 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":132,"threads_started":1}
W20260925 00:49:03.154404 10185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:03.148642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:36153 (local address 127.0.249.126:39517)
0925 00:49:03.148783 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:03.148786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:03.148937 (+ 151us) server_negotiation.cc:407] Connection header received
0925 00:49:03.149051 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:03.149054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:03.149095 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:03.149164 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:03.149642 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.150147 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.151297 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.151568 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.151646 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:03.154257 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:36153: BlockingRecv error: recv got EOF from 127.0.249.66:36153 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 00:49:03 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:03 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:03.339461 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:03.333788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:33061)
0925 00:49:03.334088 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:03.334102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:03.334236 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:03.334524 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:03.334527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:03.334733 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:03.334824 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:03.334830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.335617 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.335622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:03.336186 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:03.336193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.336701 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.336705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:03.336804 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:03.337487 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:03.337499 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:03.339347 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":90,"threads_started":1}
W20260925 00:49:03.339605 10185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:03.333877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:33061 (local address 127.0.249.126:39517)
0925 00:49:03.334037 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:03.334041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:03.334180 (+ 139us) server_negotiation.cc:407] Connection header received
0925 00:49:03.334351 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:03.334355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:03.334410 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:03.334516 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:03.334969 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.335497 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.336300 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.336572 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.336680 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:03.339488 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:33061: BlockingRecv error: recv got EOF from 127.0.249.67:33061 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 00:49:03 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:03 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:03.995060 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:03.989825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:44225)
0925 00:49:03.990081 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:03.990093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:03.990175 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:03.990402 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:03.990405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:03.990585 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:03.990676 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:03.990681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.991396 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.991398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:03.991948 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:03.991955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.992457 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.992460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:03.992530 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:03.992997 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:03.993008 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:03.994875 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
W20260925 00:49:03.995121 10189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:03.989913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:44225 (local address 127.0.249.126:39517)
0925 00:49:03.990164 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:03.990167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:03.990179 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:49:03.990254 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:03.990257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:03.990297 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:03.990363 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:03.990777 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.991306 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.992051 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:03.992268 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:03.992372 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:03.995001 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:44225: BlockingRecv error: recv got EOF from 127.0.249.65:44225 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":106,"threads_started":1}
Sep 25 00:49:04 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:04 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:04.160406 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:04.155144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:56577)
0925 00:49:04.155420 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:04.155434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:04.155523 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:04.155751 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:04.155754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:04.155971 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:04.156067 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:04.156072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.156757 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.156759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:04.157343 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:04.157352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.157808 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.157811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:04.157899 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:04.158451 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:04.158463 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:04.160275 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
W20260925 00:49:04.160530 10189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:04.155299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:56577 (local address 127.0.249.126:39517)
0925 00:49:04.155432 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:04.155436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:04.155501 (+ 65us) server_negotiation.cc:407] Connection header received
0925 00:49:04.155601 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:04.155603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:04.155643 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:04.155715 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:04.156164 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.156666 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.157507 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.157715 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.157792 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:04.160408 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:56577: BlockingRecv error: recv got EOF from 127.0.249.66:56577 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 00:49:04 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:04 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:04.345480 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:04.340321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:47023)
0925 00:49:04.340611 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:04.340625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:04.340720 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:04.340943 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:04.340945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:04.341125 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:04.341218 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:04.341223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.341974 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.341976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:04.342536 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:04.342544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.342937 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.342940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:04.343000 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:04.343515 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:04.343526 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:04.345335 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
W20260925 00:49:04.345650 10189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:04.340457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:47023 (local address 127.0.249.126:39517)
0925 00:49:04.340637 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:04.340641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:04.340691 (+ 50us) server_negotiation.cc:407] Connection header received
0925 00:49:04.340799 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:04.340801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:04.340843 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:04.340908 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:04.341311 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.341884 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.342633 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.342842 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.342922 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:04.345522 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:47023: BlockingRecv error: recv got EOF from 127.0.249.67:47023 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 00:49:04 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:04 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:05.001670 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:04.995938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:42183)
0925 00:49:04.996211 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:04.996226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:04.996330 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:04.996632 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:04.996635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:04.996879 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:04.997013 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:04.997020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.997897 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.997901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:04.998469 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:04.998476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.998956 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.998959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:04.999049 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:04.999538 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:04.999549 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:05.001498 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
W20260925 00:49:05.001776 10193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:04.995991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:42183 (local address 127.0.249.126:39517)
0925 00:49:04.996278 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:04.996282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:04.996296 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:49:04.996451 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:04.996455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:04.996503 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:04.996606 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:04.997140 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.997772 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.998593 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:04.998838 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:04.998955 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:05.001639 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:42183: BlockingRecv error: recv got EOF from 127.0.249.65:42183 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
Sep 25 00:49:05 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:05 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:05.166788 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:05.161267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:55377)
0925 00:49:05.161555 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:05.161568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:05.161696 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:05.161949 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:05.161952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:05.162160 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:05.162252 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:05.162258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:05.163016 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:05.163019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:05.163579 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:05.163586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:05.164030 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:05.164034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:05.164111 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:05.164650 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:05.164661 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:05.166613 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
W20260925 00:49:05.166850 10193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:05.161324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:55377 (local address 127.0.249.126:39517)
0925 00:49:05.161520 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:05.161525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:05.161637 (+ 112us) server_negotiation.cc:407] Connection header received
0925 00:49:05.161792 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:05.161795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:05.161847 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:05.161933 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:05.162374 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:05.162910 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:05.163695 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:05.163920 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:05.164035 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:05.166744 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:55377: BlockingRecv error: recv got EOF from 127.0.249.66:55377 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 00:49:05 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:05 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:05.351698 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:05.346354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:48797)
0925 00:49:05.346644 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:05.346656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:05.346783 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:05.347043 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:05.347046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:05.347263 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:05.347348 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:05.347353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:05.348091 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:05.348096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:05.348679 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:05.348686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:05.349124 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:05.349127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:05.349210 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:05.349785 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:05.349796 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:05.351531 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
W20260925 00:49:05.351753 10193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:05.346400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:48797 (local address 127.0.249.126:39517)
0925 00:49:05.346551 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:05.346555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:05.346724 (+ 169us) server_negotiation.cc:407] Connection header received
0925 00:49:05.346877 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:05.346880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:05.346933 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:05.347024 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:05.347456 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:05.347968 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:05.348791 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:05.349017 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:05.349124 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:05.351646 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:48797: BlockingRecv error: recv got EOF from 127.0.249.67:48797 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 00:49:06 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:06 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:06.008348 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:06.002522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:41437)
0925 00:49:06.002863 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:06.002877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:06.003014 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:06.003255 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:06.003257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:06.003435 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:06.003529 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:06.003534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.004470 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.004474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:06.005380 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:06.005391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.005853 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.005855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:06.005939 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:06.006419 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:06.006432 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:06.008234 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":71,"threads_started":1}
W20260925 00:49:06.008591 10197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:06.002596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:41437 (local address 127.0.249.126:39517)
0925 00:49:06.002884 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:06.002888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:06.002980 (+ 92us) server_negotiation.cc:407] Connection header received
0925 00:49:06.003094 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:06.003097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:06.003139 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:06.003217 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:06.003627 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.004331 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.005531 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.005753 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.005833 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:06.008458 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:41437: BlockingRecv error: recv got EOF from 127.0.249.65:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":113,"threads_started":1}
Sep 25 00:49:06 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:06 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:06.173147 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:06.167676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:58107)
0925 00:49:06.167984 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:06.167997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:06.168135 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:06.168392 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:06.168395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:06.168604 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:06.168712 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:06.168719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.169525 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.169531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:06.170088 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:06.170096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.170570 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.170574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:06.170655 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:06.171124 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:06.171135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:06.172959 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
W20260925 00:49:06.173211 10197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:06.167765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:58107 (local address 127.0.249.126:39517)
0925 00:49:06.167954 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:06.167959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:06.168087 (+ 128us) server_negotiation.cc:407] Connection header received
0925 00:49:06.168234 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:06.168237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:06.168289 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:06.168373 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:06.168830 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.169430 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.170203 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.170442 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.170577 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:06.173107 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:58107: BlockingRecv error: recv got EOF from 127.0.249.66:58107 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 00:49:06 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:06 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:06.358253 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:06.352572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:59527)
0925 00:49:06.352936 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:06.352951 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:06.353087 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:06.353373 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:06.353377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:06.353640 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:06.353736 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:06.353743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.354760 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.354763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:06.355312 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:06.355319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.355729 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.355732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:06.355791 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:06.356278 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:06.356288 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:06.358138 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":72,"threads_started":1}
W20260925 00:49:06.358395 10197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:06.352751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:59527 (local address 127.0.249.126:39517)
0925 00:49:06.352929 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:06.352934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:06.353030 (+ 96us) server_negotiation.cc:407] Connection header received
0925 00:49:06.353191 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:06.353193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:06.353242 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:06.353345 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:06.353882 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.354654 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.355418 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:06.355633 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:06.355711 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:06.358278 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:59527: BlockingRecv error: recv got EOF from 127.0.249.67:59527 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 00:49:07 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:07 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:07.014662 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:07.009126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:50635)
0925 00:49:07.009472 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:07.009486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:07.009589 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:07.009819 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:07.009820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:07.009990 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:07.010080 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:07.010085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.010812 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.010814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:07.011368 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:07.011375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.011783 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.011786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:07.011846 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:07.012323 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:07.012334 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:07.014506 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":114,"threads_started":1}
W20260925 00:49:07.014784 10201 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:07.009304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:50635 (local address 127.0.249.126:39517)
0925 00:49:07.009574 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:07.009578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:07.009597 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:49:07.009663 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:07.009665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:07.009712 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:07.009789 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:07.010180 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.010722 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.011465 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.011688 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.011768 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:07.014655 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:50635: BlockingRecv error: recv got EOF from 127.0.249.65:50635 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":123,"threads_started":1}
Sep 25 00:49:07 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:07 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:07.179452 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:07.174075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:53881)
0925 00:49:07.174328 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:07.174346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:07.174495 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:07.174803 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:07.174806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:07.175056 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:07.175182 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:07.175188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.176054 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.176056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:07.176602 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:07.176609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.176992 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.176995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:07.177054 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:07.177595 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:07.177606 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:07.179322 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
W20260925 00:49:07.179582 10201 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:07.174125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:53881 (local address 127.0.249.126:39517)
0925 00:49:07.174268 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:07.174274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:07.174447 (+ 173us) server_negotiation.cc:407] Connection header received
0925 00:49:07.174603 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:07.174609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:07.174680 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:07.174746 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:07.175286 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.175963 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.176695 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.176902 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.176974 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:07.179451 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:53881: BlockingRecv error: recv got EOF from 127.0.249.66:53881 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 00:49:07 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:07 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:07.366019 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:07.359172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:57717)
0925 00:49:07.359488 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:07.359503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:07.359645 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:07.360056 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:07.360061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:07.360333 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:07.360470 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:07.360477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.361318 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.361322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:07.362079 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:07.362090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.362879 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.362883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:07.362958 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:07.363618 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:07.363633 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:07.365889 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
W20260925 00:49:07.366851 10201 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:07.359240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:57717 (local address 127.0.249.126:39517)
0925 00:49:07.359412 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:07.359417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:07.359579 (+ 162us) server_negotiation.cc:407] Connection header received
0925 00:49:07.359752 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:07.359756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:07.359836 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:07.359969 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:07.360586 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.361190 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.362228 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:07.362687 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:07.362847 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:07.366717 (+ 3870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:57717: BlockingRecv error: recv got EOF from 127.0.249.67:57717 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 00:49:08 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:08 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:08.026465 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:08.015514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:35721)
0925 00:49:08.015823 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:08.015837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:08.015933 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:08.016185 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:08.016187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:08.016359 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:08.016466 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:08.016472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.017231 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.017233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:08.017850 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:08.017858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.018271 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.018273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:08.018338 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:08.024233 (+ 5895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:08.024247 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:08.026300 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":88,"threads_started":1}
W20260925 00:49:08.026579 10205 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:08.015638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:35721 (local address 127.0.249.126:39517)
0925 00:49:08.015871 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:08.015875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:08.015890 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:49:08.016026 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:08.016028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:08.016071 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:08.016149 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:08.016609 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.017135 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.017952 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.018175 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.018251 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:08.026445 (+ 8194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:35721: BlockingRecv error: recv got EOF from 127.0.249.65:35721 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
Sep 25 00:49:08 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:08 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:08.186331 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:08.180273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:60135)
0925 00:49:08.180579 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:08.180600 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:08.180750 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:08.181080 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:08.181084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:08.181319 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:08.181514 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:08.181525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.182453 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.182456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:08.183180 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:08.183190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.183592 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.183595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:08.183660 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:08.184296 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:08.184311 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:08.186191 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":110,"threads_started":1}
W20260925 00:49:08.186410 10205 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:08.180327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:60135 (local address 127.0.249.126:39517)
0925 00:49:08.180517 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:08.180522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:08.180696 (+ 174us) server_negotiation.cc:407] Connection header received
0925 00:49:08.180890 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:08.180893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:08.180953 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:08.181053 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:08.181654 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.182350 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.183279 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.183491 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.183570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:08.186322 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:60135: BlockingRecv error: recv got EOF from 127.0.249.66:60135 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 00:49:08 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:08 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:08.372521 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:08.366818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:42157)
0925 00:49:08.367090 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:08.367103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:08.367240 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:08.367527 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:08.367530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:08.367729 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:08.367819 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:08.367842 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.368630 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.368635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:08.369212 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:08.369219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.369739 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.369743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:08.369828 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:08.370296 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:08.370307 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:08.372317 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
W20260925 00:49:08.372543 10205 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:08.366861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:42157 (local address 127.0.249.126:39517)
0925 00:49:08.367018 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:08.367023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:08.367182 (+ 159us) server_negotiation.cc:407] Connection header received
0925 00:49:08.367355 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:08.367358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:08.367410 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:08.367494 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:08.367950 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.368521 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.369329 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:08.369628 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:08.369737 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:08.372439 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:42157: BlockingRecv error: recv got EOF from 127.0.249.67:42157 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 00:49:09 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:09 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:09.032982 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:09.027372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:53699)
0925 00:49:09.027650 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:09.027665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:09.027752 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:09.028005 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:09.028007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:09.028178 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:09.028271 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:09.028277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.029142 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.029144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:09.029760 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:09.029769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.030253 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.030256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:09.030318 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:09.030845 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:09.030864 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:09.032834 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
W20260925 00:49:09.033157 10209 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:09.027403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:53699 (local address 127.0.249.126:39517)
0925 00:49:09.027689 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:09.027693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:09.027711 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:49:09.027840 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:09.027842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:09.027884 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:09.027966 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:09.028381 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.029041 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.029905 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.030155 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.030243 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:09.032995 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:53699: BlockingRecv error: recv got EOF from 127.0.249.65:53699 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
Sep 25 00:49:09 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:09 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:09.192202 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:09.187180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:33911)
0925 00:49:09.187486 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:09.187509 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:09.187597 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:09.187819 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:09.187822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:09.187995 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:09.188087 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:09.188092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.188816 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.188818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:09.189352 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:09.189358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.189834 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.189836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:09.189898 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:09.190352 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:09.190363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:09.192094 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":127,"threads_started":1}
W20260925 00:49:09.192334 10209 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:09.187232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:33911 (local address 127.0.249.126:39517)
0925 00:49:09.187400 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:09.187405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:09.187579 (+ 174us) server_negotiation.cc:407] Connection header received
0925 00:49:09.187665 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:09.187667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:09.187713 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:09.187788 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:09.188211 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.188726 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.189524 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.189738 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.189816 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:09.192220 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:33911: BlockingRecv error: recv got EOF from 127.0.249.66:33911 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 00:49:09 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:09 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:09.379501 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:09.373380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:55815)
0925 00:49:09.373674 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:09.373688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:09.373770 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:09.374062 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:09.374065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:09.374301 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:09.374419 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:09.374432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.375503 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.375507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:09.376400 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:09.376410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.377059 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.377063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:09.377169 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:09.377676 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:09.377688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:09.379391 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":109,"threads_started":1}
W20260925 00:49:09.379642 10209 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:09.373623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:55815 (local address 127.0.249.126:39517)
0925 00:49:09.373766 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:09.373770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:09.373783 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:49:09.373887 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:09.373889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:09.373933 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:09.373999 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:09.374549 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.375375 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.376524 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:09.376956 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:09.377059 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:09.379524 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:55815: BlockingRecv error: recv got EOF from 127.0.249.67:55815 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 00:49:10 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:10 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:10.039546 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:10.034061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:50179)
0925 00:49:10.034332 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:10.034346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:10.034459 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:10.034737 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:10.034739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:10.035011 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:10.035137 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:10.035144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.035882 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.035884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:10.036425 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:10.036433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.036824 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.036826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:10.036883 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:10.037369 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:10.037380 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:10.039364 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
W20260925 00:49:10.039785 10213 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:10.034145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:50179 (local address 127.0.249.126:39517)
0925 00:49:10.034403 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:10.034407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:10.034422 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:49:10.034536 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:10.034539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:10.034589 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:10.034673 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:10.035244 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.035790 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.036517 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.036732 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.036806 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:10.039512 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:50179: BlockingRecv error: recv got EOF from 127.0.249.65:50179 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":96,"threads_started":1}
Sep 25 00:49:10 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:10 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:10.198139 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:10.192942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:44697)
0925 00:49:10.193201 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:10.193214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:10.193315 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:10.193560 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:10.193562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:10.193735 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:10.193829 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:10.193833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.194535 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.194538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:10.195095 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:10.195104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.195503 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.195506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:10.195566 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:10.196142 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:10.196154 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:10.198030 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":66,"threads_started":1}
W20260925 00:49:10.198315 10213 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:10.193070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:44697 (local address 127.0.249.126:39517)
0925 00:49:10.193223 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:10.193226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:10.193285 (+ 59us) server_negotiation.cc:407] Connection header received
0925 00:49:10.193413 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:10.193415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:10.193457 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:10.193526 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:10.193924 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.194443 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.195195 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.195411 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.195484 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:10.198158 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:44697: BlockingRecv error: recv got EOF from 127.0.249.66:44697 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 00:49:10 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:10 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:10.385617 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:10.380266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:35617)
0925 00:49:10.380554 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:10.380566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:10.380658 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:10.380882 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:10.380885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:10.381056 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:10.381147 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:10.381152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.381872 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.381874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:10.382430 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:10.382438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.382918 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.382921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:10.382985 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:10.383565 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:10.383579 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:10.385484 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
W20260925 00:49:10.385799 10213 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:10.380312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:35617 (local address 127.0.249.126:39517)
0925 00:49:10.380458 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:10.380461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:10.380636 (+ 175us) server_negotiation.cc:407] Connection header received
0925 00:49:10.380736 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:10.380738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:10.380780 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:10.380848 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:10.381243 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.381782 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.382563 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:10.382802 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:10.382898 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:10.385603 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:35617: BlockingRecv error: recv got EOF from 127.0.249.67:35617 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 00:49:11 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:11 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:11.045760 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:11.040324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:58969)
0925 00:49:11.040584 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:11.040597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:11.040683 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:11.040923 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:11.040925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:11.041108 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:11.041202 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:11.041207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.041990 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.041993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:11.042555 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:11.042562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.042981 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.042983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:11.043057 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:11.043531 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:11.043542 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:11.045618 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
W20260925 00:49:11.045876 10217 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:11.040426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:58969 (local address 127.0.249.126:39517)
0925 00:49:11.040653 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:11.040657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:11.040669 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:49:11.040766 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:11.040768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:11.040810 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:11.040919 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:11.041298 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.041869 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.042650 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.042864 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.042960 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:11.045734 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:58969: BlockingRecv error: recv got EOF from 127.0.249.65:58969 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
Sep 25 00:49:11 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:11 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:11.204339 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:11.199011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:60473)
0925 00:49:11.199287 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:11.199300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:11.199395 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:11.199606 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:11.199609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:11.199781 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:11.199875 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:11.199881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.200593 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.200596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:11.201142 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:11.201149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.201607 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.201609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:11.201670 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:11.202244 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:11.202256 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:11.204167 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":111,"threads_started":1}
W20260925 00:49:11.204394 10217 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:11.199072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:60473 (local address 127.0.249.126:39517)
0925 00:49:11.199225 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:11.199228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:11.199368 (+ 140us) server_negotiation.cc:407] Connection header received
0925 00:49:11.199475 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:11.199477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:11.199526 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:11.199594 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:11.199971 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.200472 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.201237 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.201517 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.201591 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:11.204287 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:60473: BlockingRecv error: recv got EOF from 127.0.249.66:60473 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 00:49:11 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:11 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:11.391575 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:11.386445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:46775)
0925 00:49:11.386706 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:11.386719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:11.386816 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:11.387047 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:11.387049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:11.387224 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:11.387319 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:11.387323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.388016 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.388018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:11.388558 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:11.388564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.388990 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.388993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:11.389055 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:11.389702 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:11.389714 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:11.391462 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
W20260925 00:49:11.391785 10217 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:11.386475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:46775 (local address 127.0.249.126:39517)
0925 00:49:11.386617 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:11.386620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:11.386788 (+ 168us) server_negotiation.cc:407] Connection header received
0925 00:49:11.386895 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:11.386898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:11.386942 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:11.387013 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:11.387414 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.387928 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.388655 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:11.388860 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:11.388933 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:11.391595 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:46775: BlockingRecv error: recv got EOF from 127.0.249.67:46775 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 00:49:12 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:12 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:12.051774 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:12.046535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:53895)
0925 00:49:12.046796 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:12.046809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:12.046895 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:12.047136 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:12.047138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:12.047311 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:12.047402 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:12.047407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.048196 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.048198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:12.048757 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:12.048764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.049207 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.049209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:12.049265 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:12.049785 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:12.049796 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:12.051660 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
W20260925 00:49:12.051964 10221 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:12.046594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:53895 (local address 127.0.249.126:39517)
0925 00:49:12.046835 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:12.046838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:12.046851 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:49:12.046981 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:12.046983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:12.047026 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:12.047101 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:12.047535 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.048089 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.048873 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.049106 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.049197 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:12.051794 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:53895: BlockingRecv error: recv got EOF from 127.0.249.65:53895 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":60,"threads_started":1}
Sep 25 00:49:12 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:12 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:12.211202 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:12.205282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:49161)
0925 00:49:12.205560 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:12.205578 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:12.205708 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:12.206025 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:12.206028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:12.206271 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:12.206403 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:12.206413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.207274 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.207277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:12.207839 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:12.207846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.208240 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.208243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:12.208304 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:12.208877 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:12.208897 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:12.211087 (+ 2190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
W20260925 00:49:12.211592 10221 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:12.205561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:49161 (local address 127.0.249.126:39517)
0925 00:49:12.205704 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:12.205708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:12.205722 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:49:12.205803 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:12.205806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:12.205859 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:12.205963 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:12.206545 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.207175 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.207939 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.208148 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.208223 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:12.211464 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:49161: BlockingRecv error: recv got EOF from 127.0.249.66:49161 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 00:49:12 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:12 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:12.398026 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:12.392318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:46899)
0925 00:49:12.392594 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:12.392608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:12.392940 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:12.393234 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:12.393237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:12.393540 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:12.393636 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:12.393642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.394393 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.394398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:12.394995 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:12.395003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.395455 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.395458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:12.395544 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:12.396101 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:12.396115 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:12.397892 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
W20260925 00:49:12.398234 10221 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:12.392376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:46899 (local address 127.0.249.126:39517)
0925 00:49:12.392532 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:12.392537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:12.392865 (+ 328us) server_negotiation.cc:407] Connection header received
0925 00:49:12.393064 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:12.393069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:12.393123 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:12.393210 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:12.393755 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.394286 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.395116 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:12.395338 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:12.395454 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:12.398020 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:46899: BlockingRecv error: recv got EOF from 127.0.249.67:46899 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 00:49:13 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:13 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:13.063879 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:13.052448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:60407)
0925 00:49:13.052698 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:13.052718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:13.052880 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:13.053113 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:13.053115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:13.053295 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:13.053443 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:13.053458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.054196 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.054199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:13.054764 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:13.054772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.055200 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.055203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:13.055269 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:13.061813 (+ 6544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:13.061825 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:13.063726 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
W20260925 00:49:13.064023 10225 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:13.052589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:60407 (local address 127.0.249.126:39517)
0925 00:49:13.052894 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:13.052897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:13.052910 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:49:13.052948 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:13.052950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:13.052990 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:13.053075 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:13.053558 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.054075 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.054885 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.055103 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.055176 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:13.063866 (+ 8690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:60407: BlockingRecv error: recv got EOF from 127.0.249.65:60407 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":79,"threads_started":1}
Sep 25 00:49:13 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:13 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:13.217623 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:13.211982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:44703)
0925 00:49:13.212232 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:13.212246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:13.212350 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:13.212571 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:13.212573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:13.212751 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:13.212881 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:13.212888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.213669 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.213672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:13.214269 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:13.214278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.214719 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.214721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:13.214859 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:13.215485 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:13.215499 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:13.217486 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
W20260925 00:49:13.217747 10225 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:13.212124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:44703 (local address 127.0.249.126:39517)
0925 00:49:13.212280 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:13.212284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:13.212296 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:49:13.212428 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:13.212431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:13.212472 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:13.212531 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:13.212997 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.213557 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.214390 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.214618 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.214711 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:13.217632 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:44703: BlockingRecv error: recv got EOF from 127.0.249.66:44703 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 00:49:13 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:13 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.67@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:13.404009 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:13.398736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.67:55617)
0925 00:49:13.399010 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:13.399024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:13.399196 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:13.399471 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:13.399474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:13.399682 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:13.399773 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:13.399778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.400500 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.400505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:13.401070 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:13.401078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.401570 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.401573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:13.401659 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:13.402218 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:13.402228 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:13.403880 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
W20260925 00:49:13.404210 10225 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:13.398800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:55617 (local address 127.0.249.126:39517)
0925 00:49:13.398974 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:13.398978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:13.399128 (+ 150us) server_negotiation.cc:407] Connection header received
0925 00:49:13.399316 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:13.399319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:13.399363 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:13.399447 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:13.399895 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.400396 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.401189 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:13.401461 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:13.401567 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:13.404052 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.67:55617: BlockingRecv error: recv got EOF from 127.0.249.67:55617 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 00:49:14 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:14 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.65@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:14.070672 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:14.064873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.65:56917)
0925 00:49:14.065249 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:14.065266 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:14.065426 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:14.065891 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:14.065894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:14.066139 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:14.066302 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:14.066313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.067118 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.067121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:14.067728 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:14.067737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.068227 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.068230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:14.068317 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:14.068791 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:14.068802 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:14.070550 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
W20260925 00:49:14.070806 10229 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:14.064981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:56917 (local address 127.0.249.126:39517)
0925 00:49:14.065256 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:14.065266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:14.065327 (+ 61us) server_negotiation.cc:407] Connection header received
0925 00:49:14.065555 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:14.065560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:14.065622 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:14.065712 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:14.066423 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.067031 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.067824 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.068131 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.068212 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:14.070673 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.65:56917: BlockingRecv error: recv got EOF from 127.0.249.65:56917 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
Sep 25 00:49:14 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 00:49:14 dist-test-slave-05m0 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.249.66@KRBTEST.COM for oryx/127.0.249.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 00:49:14.223932 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:14.218578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39517 (local address 127.0.249.66:41379)
0925 00:49:14.218838 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:14.218851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:14.218941 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:14.219163 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:14.219165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:14.219347 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:14.219440 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:14.219446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.220191 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.220194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:14.220793 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:14.220801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.221242 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.221246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:14.221316 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:14.221938 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:14.221950 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:14.223818 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.249.126:39517: Server oryx/127.0.249.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
W20260925 00:49:14.224067 10229 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:14.218666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:41379 (local address 127.0.249.126:39517)
0925 00:49:14.218801 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:14.218804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:14.218916 (+ 112us) server_negotiation.cc:407] Connection header received
0925 00:49:14.219018 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:14.219020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:14.219061 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:14.219124 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:14.219547 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.220061 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.220902 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.221117 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.221193 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:14.223947 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.249.66:41379: BlockingRecv error: recv got EOF from 127.0.249.66:41379 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:49:14.297040 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 9753
I20260925 00:49:14.302829 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 9905
I20260925 00:49:14.307937 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 10038
I20260925 00:49:14.312958 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 9683
2026-09-25T00:49:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17964 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T00:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:49:14Z Disabled control of system clock
I20260925 00:49:14.343143 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:35045
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:35045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 00:49:14.452253 10234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:14.452527 10234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:14.452565 10234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:14.456698 10234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:49:14.456758 10234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:14.456774 10234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:14.456791 10234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:49:14.456811 10234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:49:14.461596 10234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:35045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:35045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:14.462657 10234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:14.463992 10234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:14.472708 10240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:14.472748 10234 server_base.cc:1034] running on GCE node
W20260925 00:49:14.472802 10242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:14.472939 10239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:14.473558 10234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:14.474610 10234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:14.475767 10234 hybrid_clock.cc:648] HybridClock initialized: now 1790297354475733 us; error 47 us; skew 500 ppm
I20260925 00:49:14.477705 10234 webserver.cc:460] Webserver started at http://127.0.249.126:38995/ using document root <none> and password file <none>
I20260925 00:49:14.478268 10234 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:14.478336 10234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:14.478539 10234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:14.480767 10234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "6bf2011ef7d842ee813d4a066fa39b58"
format_stamp: "Formatted at 2026-09-25 00:49:14 on dist-test-slave-05m0"
server_key: "cab08b749dccb8c3c5b09726ca124c0f"
server_key_iv: "88293547aee841b2338b1113afb20495"
server_key_version: "encryptionkey@0"
I20260925 00:49:14.481479 10234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "6bf2011ef7d842ee813d4a066fa39b58"
format_stamp: "Formatted at 2026-09-25 00:49:14 on dist-test-slave-05m0"
server_key: "cab08b749dccb8c3c5b09726ca124c0f"
server_key_iv: "88293547aee841b2338b1113afb20495"
server_key_version: "encryptionkey@0"
I20260925 00:49:14.484982 10234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:49:14.487426 10248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:14.488678 10234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:49:14.488826 10234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "6bf2011ef7d842ee813d4a066fa39b58"
format_stamp: "Formatted at 2026-09-25 00:49:14 on dist-test-slave-05m0"
server_key: "cab08b749dccb8c3c5b09726ca124c0f"
server_key_iv: "88293547aee841b2338b1113afb20495"
server_key_version: "encryptionkey@0"
I20260925 00:49:14.488976 10234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:14.500474 10234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:14.501137 10234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:14.501327 10234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:14.507959 10300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:35045 every 8 connection(s)
I20260925 00:49:14.507958 10234 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:35045
I20260925 00:49:14.508884 10234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:49:14.511817 10301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:49:14.517033 10301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58: Bootstrap starting.
I20260925 00:49:14.518659 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 10234
I20260925 00:49:14.518790 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:49:14.519068 997 external_mini_cluster.cc:1428] Setting key e09aa15eb7e692e9ef9abd0ce0386625
I20260925 00:49:14.519568 10301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58: Neither blocks nor log segments found. Creating new log.
I20260925 00:49:14.520412 10301 log.cc:824] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58: Log is configured to *not* fsync() on all Append() calls
I20260925 00:49:14.522423 10301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58: No bootstrap required, opened a new log
I20260925 00:49:14.526640 10301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2011ef7d842ee813d4a066fa39b58" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35045 } }
I20260925 00:49:14.526851 10301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:49:14.526932 10301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf2011ef7d842ee813d4a066fa39b58, State: Initialized, Role: FOLLOWER
I20260925 00:49:14.527431 10301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2011ef7d842ee813d4a066fa39b58" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35045 } }
I20260925 00:49:14.527593 10301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:49:14.527657 10301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:49:14.527755 10301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:49:14.528398 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:14.520225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.0.249.126:35045)
0925 00:49:14.520922 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:14.520940 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:14.520983 (+ 43us) server_negotiation.cc:407] Connection header received
0925 00:49:14.521737 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:14.521754 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:14.522080 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:14.522347 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:14.522729 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.523657 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.524897 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.525349 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.525567 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:14.526464 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:14.526469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:14.526481 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:49:14.526890 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:14.527013 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:49:14.527045 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:14.527119 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:14.527292 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 00:49:14.527585 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":164,"threads_started":1}
I20260925 00:49:14.528697 10301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2011ef7d842ee813d4a066fa39b58" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35045 } }
I20260925 00:49:14.529039 10301 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf2011ef7d842ee813d4a066fa39b58; no voters:
I20260925 00:49:14.529348 10301 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:49:14.529451 10306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:49:14.529632 10306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [term 1 LEADER]: Becoming Leader. State: Replica: 6bf2011ef7d842ee813d4a066fa39b58, State: Running, Role: LEADER
I20260925 00:49:14.529971 10306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2011ef7d842ee813d4a066fa39b58" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35045 } }
I20260925 00:49:14.530618 10301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:49:14.531692 10308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf2011ef7d842ee813d4a066fa39b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2011ef7d842ee813d4a066fa39b58" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35045 } } }
I20260925 00:49:14.531814 10308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [sys.catalog]: This master's current role is: LEADER
I20260925 00:49:14.531956 10307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf2011ef7d842ee813d4a066fa39b58. Latest consensus state: current_term: 1 leader_uuid: "6bf2011ef7d842ee813d4a066fa39b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2011ef7d842ee813d4a066fa39b58" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35045 } } }
I20260925 00:49:14.532135 10307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58 [sys.catalog]: This master's current role is: LEADER
I20260925 00:49:14.532414 10315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:49:14.535482 10315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:49:14.540601 10315 catalog_manager.cc:1347] Generated new cluster ID: 81defc74209e404c8767d79b899ea77f
I20260925 00:49:14.540664 10315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:49:14.561767 10315 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:49:14.562897 10315 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:49:14.570503 10315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf2011ef7d842ee813d4a066fa39b58: Generated new TSK 0
I20260925 00:49:14.571184 10315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:49:14.576210 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:35045
--builtin_ntp_servers=127.0.249.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 00:49:14.697484 10325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:14.697717 10325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:14.697752 10325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:14.701723 10325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:14.701779 10325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:14.701874 10325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:49:14.706735 10325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:14.707819 10325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:14.709196 10325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:14.718755 10333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:14.718823 10331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:14.718819 10330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:14.719122 10325 server_base.cc:1034] running on GCE node
I20260925 00:49:14.719539 10325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:14.720237 10325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:14.721482 10325 hybrid_clock.cc:648] HybridClock initialized: now 1790297354721462 us; error 29 us; skew 500 ppm
I20260925 00:49:14.723366 10325 webserver.cc:460] Webserver started at http://127.0.249.65:44415/ using document root <none> and password file <none>
I20260925 00:49:14.723963 10325 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:14.724056 10325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:14.724288 10325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:14.726387 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "39a7c4c4cdc340bf84ac6671b1497499"
format_stamp: "Formatted at 2026-09-25 00:49:14 on dist-test-slave-05m0"
server_key: "2796954e2eef1989bfda9d14e1c40504"
server_key_iv: "f59b2039879b262a4cae8fb7463c6ab4"
server_key_version: "encryptionkey@0"
I20260925 00:49:14.726969 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "39a7c4c4cdc340bf84ac6671b1497499"
format_stamp: "Formatted at 2026-09-25 00:49:14 on dist-test-slave-05m0"
server_key: "2796954e2eef1989bfda9d14e1c40504"
server_key_iv: "f59b2039879b262a4cae8fb7463c6ab4"
server_key_version: "encryptionkey@0"
I20260925 00:49:14.731109 10325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:49:14.733891 10339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:14.735189 10325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:49:14.735316 10325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "39a7c4c4cdc340bf84ac6671b1497499"
format_stamp: "Formatted at 2026-09-25 00:49:14 on dist-test-slave-05m0"
server_key: "2796954e2eef1989bfda9d14e1c40504"
server_key_iv: "f59b2039879b262a4cae8fb7463c6ab4"
server_key_version: "encryptionkey@0"
I20260925 00:49:14.735432 10325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:14.751693 10325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:14.752449 10325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:14.752700 10325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:14.753464 10325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:49:14.754488 10325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:49:14.754565 10325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:14.754637 10325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:49:14.754693 10325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:14.764253 10325 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:41731
I20260925 00:49:14.764276 10452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:41731 every 8 connection(s)
I20260925 00:49:14.765216 10325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:49:14.773537 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 10325
I20260925 00:49:14.773660 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:49:14.773998 997 external_mini_cluster.cc:1428] Setting key 0dbcbf6404c533a395f0b73ecbee2f2e
I20260925 00:49:14.774421 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:14.767069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:45879 (local address 127.0.249.126:35045)
0925 00:49:14.767196 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:14.767200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:14.768248 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 00:49:14.769677 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:14.769680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:14.769741 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:14.769820 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:14.770694 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.771254 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.772370 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.772626 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.772753 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:14.773942 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:14.773945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:14.773947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:49:14.773959 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:14.773978 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:49:14.773986 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:14.774074 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:14.774236 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 00:49:14.774290 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:49:14.775274 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:14.767471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:35045 (local address 127.0.249.65:45879)
0925 00:49:14.768095 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:14.768133 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:14.769454 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:14.769986 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:14.769994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:14.770073 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:14.770546 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:14.770557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.771385 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.771389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:14.772240 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:14.772249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.772790 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.772794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:14.772954 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:14.773625 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:49:14.773627 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:14.773684 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:49:14.773762 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:49:14.774080 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:49:14.774091 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:49:14.774097 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:49:14.774229 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 00:49:14.774527 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":123,"threads_started":1}
I20260925 00:49:14.776011 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:35045
--builtin_ntp_servers=127.0.249.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 00:49:14.776494 10453 heartbeater.cc:347] Connected to a master server at 127.0.249.126:35045
I20260925 00:49:14.776721 10453 heartbeater.cc:464] Registering TS with master...
I20260925 00:49:14.777320 10453 heartbeater.cc:511] Master 127.0.249.126:35045 requested a full tablet report, sending...
I20260925 00:49:14.778899 10265 ts_manager.cc:194] Registered new tserver with Master: 39a7c4c4cdc340bf84ac6671b1497499 (127.0.249.65:41731)
I20260925 00:49:14.780093 10265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.65:45879
W20260925 00:49:14.883733 10456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:14.883963 10456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:14.883998 10456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:14.888003 10456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:14.888052 10456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:14.888137 10456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:49:14.892746 10456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:14.893862 10456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:14.895182 10456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:14.904364 10464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:14.904361 10461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:14.904655 10462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:14.904721 10456 server_base.cc:1034] running on GCE node
I20260925 00:49:14.905144 10456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:14.905831 10456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:14.907023 10456 hybrid_clock.cc:648] HybridClock initialized: now 1790297354907012 us; error 26 us; skew 500 ppm
I20260925 00:49:14.908843 10456 webserver.cc:460] Webserver started at http://127.0.249.66:37547/ using document root <none> and password file <none>
I20260925 00:49:14.909461 10456 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:14.909549 10456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:14.909771 10456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:14.911834 10456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "8e8ccd2c1a5f47c783bfd9672fc1b305"
format_stamp: "Formatted at 2026-09-25 00:49:14 on dist-test-slave-05m0"
server_key: "8ca4f87239238013e80c14ae5ad4a2b1"
server_key_iv: "3845af23c00367e8db70387be90d7649"
server_key_version: "encryptionkey@0"
I20260925 00:49:14.912402 10456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "8e8ccd2c1a5f47c783bfd9672fc1b305"
format_stamp: "Formatted at 2026-09-25 00:49:14 on dist-test-slave-05m0"
server_key: "8ca4f87239238013e80c14ae5ad4a2b1"
server_key_iv: "3845af23c00367e8db70387be90d7649"
server_key_version: "encryptionkey@0"
I20260925 00:49:14.916113 10456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:49:14.918620 10470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:14.919773 10456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:49:14.919924 10456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "8e8ccd2c1a5f47c783bfd9672fc1b305"
format_stamp: "Formatted at 2026-09-25 00:49:14 on dist-test-slave-05m0"
server_key: "8ca4f87239238013e80c14ae5ad4a2b1"
server_key_iv: "3845af23c00367e8db70387be90d7649"
server_key_version: "encryptionkey@0"
I20260925 00:49:14.920033 10456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:14.935232 10456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:14.935842 10456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:14.936033 10456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:14.936563 10456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:49:14.937493 10456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:49:14.937556 10456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:14.937618 10456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:49:14.937668 10456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:14.946832 10456 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:45479
I20260925 00:49:14.946854 10583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:45479 every 8 connection(s)
I20260925 00:49:14.947697 10456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:49:14.952399 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 10456
I20260925 00:49:14.952519 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:49:14.952816 997 external_mini_cluster.cc:1428] Setting key a68ed2581309aa39c2263e8470fe889b
I20260925 00:49:14.954910 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:35045
--builtin_ntp_servers=127.0.249.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 00:49:14.957593 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:14.949441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:43627 (local address 127.0.249.126:35045)
0925 00:49:14.949602 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:14.949606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:14.950680 (+ 1074us) server_negotiation.cc:407] Connection header received
0925 00:49:14.951962 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:14.951964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:14.952010 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:14.952083 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:14.953264 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.953878 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.955479 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.955695 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.955770 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:14.957041 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:14.957047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:14.957054 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:49:14.957071 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:14.957096 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:49:14.957106 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:14.957193 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:14.957329 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 00:49:14.957442 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:49:14.958628 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:14.949846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:35045 (local address 127.0.249.66:43627)
0925 00:49:14.950524 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:14.950562 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:14.951785 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:14.952291 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:14.952300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:14.952411 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:14.953095 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:14.953114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.954017 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.954021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:14.955342 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:14.955356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:14.955838 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:14.955844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:14.956087 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:14.956791 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:49:14.956792 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:14.956829 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:49:14.956889 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:49:14.957199 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:49:14.957205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:49:14.957209 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 00:49:14.957300 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 00:49:14.957627 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":106,"threads_started":1}
I20260925 00:49:14.959743 10584 heartbeater.cc:347] Connected to a master server at 127.0.249.126:35045
I20260925 00:49:14.959983 10584 heartbeater.cc:464] Registering TS with master...
I20260925 00:49:14.960666 10584 heartbeater.cc:511] Master 127.0.249.126:35045 requested a full tablet report, sending...
I20260925 00:49:14.961882 10265 ts_manager.cc:194] Registered new tserver with Master: 8e8ccd2c1a5f47c783bfd9672fc1b305 (127.0.249.66:45479)
I20260925 00:49:14.962477 10265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.66:43627
W20260925 00:49:15.065552 10587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:15.065850 10587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:15.065918 10587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:15.070103 10587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:15.070174 10587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:15.070253 10587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:49:15.075008 10587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:15.076027 10587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:15.077337 10587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:49:15.086767 10587 server_base.cc:1034] running on GCE node
W20260925 00:49:15.086750 10595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:15.086735 10593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:15.086743 10592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:15.087436 10587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:15.088074 10587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:15.089237 10587 hybrid_clock.cc:648] HybridClock initialized: now 1790297355089219 us; error 28 us; skew 500 ppm
I20260925 00:49:15.091423 10587 webserver.cc:460] Webserver started at http://127.0.249.67:46027/ using document root <none> and password file <none>
I20260925 00:49:15.092127 10587 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:15.092218 10587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:15.092489 10587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:15.094735 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "dac4500915ed41ef923eb24c8c735790"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "7ec909cb1b3c9ce11432a4a6d8e60bfc"
server_key_iv: "2527567a8f3753c663562f0f7671f1bf"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.095288 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "dac4500915ed41ef923eb24c8c735790"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "7ec909cb1b3c9ce11432a4a6d8e60bfc"
server_key_iv: "2527567a8f3753c663562f0f7671f1bf"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.098728 10587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:49:15.101239 10601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.102447 10587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:49:15.102548 10587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "dac4500915ed41ef923eb24c8c735790"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "7ec909cb1b3c9ce11432a4a6d8e60bfc"
server_key_iv: "2527567a8f3753c663562f0f7671f1bf"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.102643 10587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:15.128504 10587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:15.129158 10587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:15.129303 10587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:15.129895 10587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:49:15.130957 10587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:49:15.131006 10587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.131068 10587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:49:15.131102 10587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.140131 10587 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:43393
I20260925 00:49:15.140211 10714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:43393 every 8 connection(s)
I20260925 00:49:15.141072 10587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:49:15.141911 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 10587
I20260925 00:49:15.142031 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:49:15.142302 997 external_mini_cluster.cc:1428] Setting key 54e323e13116b6cb3e188e8cf2cc21d6
I20260925 00:49:15.151100 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:15.143071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:44019 (local address 127.0.249.126:35045)
0925 00:49:15.143195 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:15.143199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:15.144919 (+ 1720us) server_negotiation.cc:407] Connection header received
0925 00:49:15.146349 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:15.146352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:15.146407 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:15.146477 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:15.147331 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:15.147881 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:15.149012 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:15.149260 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:15.149370 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:15.150591 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:15.150595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:15.150600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:49:15.150617 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:15.150635 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:49:15.150642 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:15.150718 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:15.150876 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 00:49:15.150938 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 00:49:15.151933 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:15.143491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:35045 (local address 127.0.249.67:44019)
0925 00:49:15.144791 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:15.144831 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:15.146147 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:15.146648 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:15.146655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:15.146736 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:15.147200 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:15.147212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:15.148009 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:15.148018 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:15.148885 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:15.148894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:15.149434 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:15.149439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:15.149604 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:15.150189 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:49:15.150191 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:15.150232 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:49:15.150296 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:49:15.150731 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:49:15.150740 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:49:15.150746 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:49:15.150877 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 00:49:15.151160 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260925 00:49:15.153082 10715 heartbeater.cc:347] Connected to a master server at 127.0.249.126:35045
I20260925 00:49:15.153362 10715 heartbeater.cc:464] Registering TS with master...
I20260925 00:49:15.153987 10715 heartbeater.cc:511] Master 127.0.249.126:35045 requested a full tablet report, sending...
I20260925 00:49:15.155025 10265 ts_manager.cc:194] Registered new tserver with Master: dac4500915ed41ef923eb24c8c735790 (127.0.249.67:43393)
I20260925 00:49:15.155763 10265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.67:44019
I20260925 00:49:15.156381 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 00:49:15.159106 10304 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:15.158267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.0.249.126:35045)
0925 00:49:15.158355 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:15.158358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:15.158590 (+ 232us) server_negotiation.cc:407] Connection header received
0925 00:49:15.158678 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:15.158680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:15.158717 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:15.158780 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:15.158990 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50208: BlockingRecv error: recv got EOF from 127.0.0.1:50208 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 00:49:15.159344 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 10325
I20260925 00:49:15.165426 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 10456
I20260925 00:49:15.170440 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 10587
I20260925 00:49:15.175437 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 10234
2026-09-25T00:49:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (861 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T00:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:49:15Z Disabled control of system clock
I20260925 00:49:15.204257 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:45839
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:45839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 00:49:15.315917 10726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:15.316183 10726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:15.316234 10726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:15.320425 10726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:49:15.320509 10726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:15.320534 10726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:15.320556 10726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:49:15.320577 10726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:49:15.328532 10726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:45839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:45839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:15.330050 10726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:15.331874 10726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:15.340798 10731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:15.340878 10734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:15.340804 10732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:15.341152 10726 server_base.cc:1034] running on GCE node
I20260925 00:49:15.341794 10726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:15.342904 10726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:15.344103 10726 hybrid_clock.cc:648] HybridClock initialized: now 1790297355344091 us; error 27 us; skew 500 ppm
I20260925 00:49:15.346014 10726 webserver.cc:460] Webserver started at http://127.0.249.126:34085/ using document root <none> and password file <none>
I20260925 00:49:15.346583 10726 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:15.346675 10726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:15.346901 10726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:15.348950 10726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "2f62f489504b481faf65204a9ac30bc4"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "48f8effc13042cc6f7b7140cf184eeb3"
server_key_iv: "e21ec8c62722ae9cced56fd9064ed154"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.349581 10726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "2f62f489504b481faf65204a9ac30bc4"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "48f8effc13042cc6f7b7140cf184eeb3"
server_key_iv: "e21ec8c62722ae9cced56fd9064ed154"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.353173 10726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:49:15.355515 10740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.356688 10726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:49:15.356827 10726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "2f62f489504b481faf65204a9ac30bc4"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "48f8effc13042cc6f7b7140cf184eeb3"
server_key_iv: "e21ec8c62722ae9cced56fd9064ed154"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.356936 10726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:15.361320 10726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:15.361912 10726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:15.362092 10726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:15.368643 10726 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:45839
I20260925 00:49:15.368643 10792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:45839 every 8 connection(s)
I20260925 00:49:15.369524 10726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:49:15.370507 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 10726
I20260925 00:49:15.370637 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:49:15.370910 997 external_mini_cluster.cc:1428] Setting key 62d2c5d6392e06ecdd9d3e26dbaec499
I20260925 00:49:15.372757 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:49:15.376579 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:15.372268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.0.249.126:45839)
0925 00:49:15.372951 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:15.372976 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:15.373023 (+ 47us) server_negotiation.cc:407] Connection header received
0925 00:49:15.373849 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:15.373866 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:15.374193 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:15.374539 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:15.375375 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:15.375380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:15.375391 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:49:15.375402 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:15.375458 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:49:15.375476 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:15.375633 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:15.375704 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 00:49:15.375939 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":147,"threads_started":1}
I20260925 00:49:15.378597 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4: Bootstrap starting.
I20260925 00:49:15.380532 10793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4: Neither blocks nor log segments found. Creating new log.
I20260925 00:49:15.381307 10793 log.cc:824] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4: Log is configured to *not* fsync() on all Append() calls
I20260925 00:49:15.383155 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4: No bootstrap required, opened a new log
I20260925 00:49:15.386520 10793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62f489504b481faf65204a9ac30bc4" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45839 } }
I20260925 00:49:15.386708 10793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:49:15.386740 10793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f62f489504b481faf65204a9ac30bc4, State: Initialized, Role: FOLLOWER
I20260925 00:49:15.387063 10793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62f489504b481faf65204a9ac30bc4" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45839 } }
I20260925 00:49:15.387183 10793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:49:15.387223 10793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:49:15.387282 10793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:49:15.388029 10793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62f489504b481faf65204a9ac30bc4" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45839 } }
I20260925 00:49:15.388298 10793 leader_election.cc:302] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f62f489504b481faf65204a9ac30bc4; no voters:
I20260925 00:49:15.388514 10793 leader_election.cc:288] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:49:15.388641 10798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:49:15.388883 10798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f62f489504b481faf65204a9ac30bc4, State: Running, Role: LEADER
I20260925 00:49:15.389535 10798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62f489504b481faf65204a9ac30bc4" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45839 } }
I20260925 00:49:15.389839 10793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:49:15.391220 10800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f62f489504b481faf65204a9ac30bc4. Latest consensus state: current_term: 1 leader_uuid: "2f62f489504b481faf65204a9ac30bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62f489504b481faf65204a9ac30bc4" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45839 } } }
I20260925 00:49:15.391252 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f62f489504b481faf65204a9ac30bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62f489504b481faf65204a9ac30bc4" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 45839 } } }
I20260925 00:49:15.391331 10800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [sys.catalog]: This master's current role is: LEADER
I20260925 00:49:15.391355 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4 [sys.catalog]: This master's current role is: LEADER
I20260925 00:49:15.391671 10808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:49:15.394374 10808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:49:15.399513 10808 catalog_manager.cc:1347] Generated new cluster ID: d6340fd9a74c4c4a8de3e603270f7c66
I20260925 00:49:15.399617 10808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:49:15.409446 10808 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:49:15.409747 10808 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:49:15.419281 10808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f62f489504b481faf65204a9ac30bc4: Generated new TSK 0
I20260925 00:49:15.420039 10808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:49:15.432618 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45839
--builtin_ntp_servers=127.0.249.84:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 00:49:15.553916 10817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:15.554194 10817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:15.554263 10817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:15.558293 10817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:15.558359 10817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:15.558457 10817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:49:15.563388 10817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:15.564435 10817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:15.565831 10817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:15.575282 10825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:15.575404 10817 server_base.cc:1034] running on GCE node
W20260925 00:49:15.575332 10822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:15.575300 10823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:15.575935 10817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:15.576550 10817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:15.577724 10817 hybrid_clock.cc:648] HybridClock initialized: now 1790297355577692 us; error 44 us; skew 500 ppm
I20260925 00:49:15.579378 10817 webserver.cc:460] Webserver started at http://127.0.249.65:39137/ using document root <none> and password file <none>
I20260925 00:49:15.579946 10817 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:15.580044 10817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:15.580271 10817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:15.582298 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "6c40d8b95a784730af434eecaa7fefab"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "3b609eb4131f66f59c4412f8b70c5fbf"
server_key_iv: "f03552aacd198b0651f34901f9282c2c"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.582882 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "6c40d8b95a784730af434eecaa7fefab"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "3b609eb4131f66f59c4412f8b70c5fbf"
server_key_iv: "f03552aacd198b0651f34901f9282c2c"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.586447 10817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:49:15.588794 10831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.590293 10817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:49:15.590406 10817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "6c40d8b95a784730af434eecaa7fefab"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "3b609eb4131f66f59c4412f8b70c5fbf"
server_key_iv: "f03552aacd198b0651f34901f9282c2c"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.590581 10817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:15.595177 10817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:15.595767 10817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:15.595942 10817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:15.596457 10817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:49:15.597313 10817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:49:15.597383 10817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.597476 10817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:49:15.597540 10817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.605793 10817 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:46807
I20260925 00:49:15.605860 10944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:46807 every 8 connection(s)
I20260925 00:49:15.606650 10817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:49:15.609000 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 10817
I20260925 00:49:15.609120 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:49:15.609454 997 external_mini_cluster.cc:1428] Setting key 114ab49e39354cdfb66e38d29d267595
I20260925 00:49:15.611385 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45839
--builtin_ntp_servers=127.0.249.84:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 00:49:15.612620 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:15.608398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:34143 (local address 127.0.249.126:45839)
0925 00:49:15.608543 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:15.608547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:15.609609 (+ 1062us) server_negotiation.cc:407] Connection header received
0925 00:49:15.610917 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:15.610920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:15.610960 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:15.611046 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:15.612156 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:15.612160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:15.612163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:49:15.612175 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:15.612197 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:49:15.612206 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:15.612309 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:15.612447 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 00:49:15.612491 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:49:15.613699 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:15.608716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:45839 (local address 127.0.249.65:34143)
0925 00:49:15.609446 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:15.609504 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:15.610680 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:15.611205 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:15.611212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:15.611297 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:15.611898 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:49:15.611900 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:15.611938 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:49:15.612006 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:49:15.612295 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:49:15.612304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:49:15.612309 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:49:15.612489 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 00:49:15.612727 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":139,"threads_started":1}
I20260925 00:49:15.615016 10945 heartbeater.cc:347] Connected to a master server at 127.0.249.126:45839
I20260925 00:49:15.615242 10945 heartbeater.cc:464] Registering TS with master...
I20260925 00:49:15.615689 10945 heartbeater.cc:511] Master 127.0.249.126:45839 requested a full tablet report, sending...
I20260925 00:49:15.617137 10757 ts_manager.cc:194] Registered new tserver with Master: 6c40d8b95a784730af434eecaa7fefab (127.0.249.65:46807)
W20260925 00:49:15.720065 10948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:15.720376 10948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:15.720427 10948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:15.724813 10948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:15.724875 10948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:15.724968 10948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:49:15.729624 10948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:15.730674 10948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:15.732018 10948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:15.741427 10956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:15.741667 10948 server_base.cc:1034] running on GCE node
W20260925 00:49:15.741389 10953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:15.741379 10954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:15.742127 10948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:15.742817 10948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:15.744017 10948 hybrid_clock.cc:648] HybridClock initialized: now 1790297355744001 us; error 30 us; skew 500 ppm
I20260925 00:49:15.746059 10948 webserver.cc:460] Webserver started at http://127.0.249.66:41413/ using document root <none> and password file <none>
I20260925 00:49:15.746663 10948 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:15.746757 10948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:15.746987 10948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:15.749034 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "46d22f50ca984225aa9ae5dcf53410a0"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "6ae19130e8cd4331c7f63347a6206df9"
server_key_iv: "ee23d34a9e2ee22cb1ed90d59d425d46"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.749639 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "46d22f50ca984225aa9ae5dcf53410a0"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "6ae19130e8cd4331c7f63347a6206df9"
server_key_iv: "ee23d34a9e2ee22cb1ed90d59d425d46"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.753247 10948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:49:15.755590 10962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.756768 10948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:49:15.756898 10948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "46d22f50ca984225aa9ae5dcf53410a0"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "6ae19130e8cd4331c7f63347a6206df9"
server_key_iv: "ee23d34a9e2ee22cb1ed90d59d425d46"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.757022 10948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:15.761677 10948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:15.762250 10948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:15.762434 10948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:15.762948 10948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:49:15.763790 10948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:49:15.763859 10948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.763921 10948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:49:15.763972 10948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.772542 10948 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:36415
I20260925 00:49:15.772570 11075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:36415 every 8 connection(s)
I20260925 00:49:15.773358 10948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:49:15.778147 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 10948
I20260925 00:49:15.778273 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:49:15.778585 997 external_mini_cluster.cc:1428] Setting key 40cbbb1ac2e7691beddc196d8c0a47d3
I20260925 00:49:15.779235 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:15.775079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:37445 (local address 127.0.249.126:45839)
0925 00:49:15.775239 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:15.775244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:15.776240 (+ 996us) server_negotiation.cc:407] Connection header received
0925 00:49:15.777533 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:15.777536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:15.777571 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:15.777642 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:15.778757 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:15.778761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:15.778763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:49:15.778776 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:15.778805 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:49:15.778814 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:15.778928 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:15.779063 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 00:49:15.779106 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:49:15.780114 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:15.775471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:45839 (local address 127.0.249.66:37445)
0925 00:49:15.776095 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:15.776133 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:15.777263 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:15.777795 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:15.777801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:15.777908 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:15.778488 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:49:15.778489 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:15.778526 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:49:15.778590 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:49:15.778914 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:49:15.778922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:49:15.778927 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:49:15.779105 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 00:49:15.779327 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20260925 00:49:15.780542 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45839
--builtin_ntp_servers=127.0.249.84:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 00:49:15.781282 11076 heartbeater.cc:347] Connected to a master server at 127.0.249.126:45839
I20260925 00:49:15.781528 11076 heartbeater.cc:464] Registering TS with master...
I20260925 00:49:15.781940 11076 heartbeater.cc:511] Master 127.0.249.126:45839 requested a full tablet report, sending...
I20260925 00:49:15.782938 10757 ts_manager.cc:194] Registered new tserver with Master: 46d22f50ca984225aa9ae5dcf53410a0 (127.0.249.66:36415)
W20260925 00:49:15.890422 11079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:15.890714 11079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:15.890798 11079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:15.895148 11079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:15.895213 11079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:15.895301 11079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:49:15.900187 11079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:45839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:15.901235 11079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:15.902661 11079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:15.911893 11084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:15.911888 11085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:15.911944 11087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:15.912027 11079 server_base.cc:1034] running on GCE node
I20260925 00:49:15.912575 11079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:15.913187 11079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:15.914368 11079 hybrid_clock.cc:648] HybridClock initialized: now 1790297355914345 us; error 42 us; skew 500 ppm
I20260925 00:49:15.916133 11079 webserver.cc:460] Webserver started at http://127.0.249.67:39057/ using document root <none> and password file <none>
I20260925 00:49:15.916692 11079 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:15.916785 11079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:15.917011 11079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:15.919049 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "380b599493ee4a2ba4417bb7c4fa00ea"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "0122bcd465e40bfe64580b2303ad08ce"
server_key_iv: "bcaf2c5587542ddf407afffce1a88c75"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.919591 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "380b599493ee4a2ba4417bb7c4fa00ea"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "0122bcd465e40bfe64580b2303ad08ce"
server_key_iv: "bcaf2c5587542ddf407afffce1a88c75"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.923116 11079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 00:49:15.925491 11093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.926609 11079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:49:15.926739 11079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "380b599493ee4a2ba4417bb7c4fa00ea"
format_stamp: "Formatted at 2026-09-25 00:49:15 on dist-test-slave-05m0"
server_key: "0122bcd465e40bfe64580b2303ad08ce"
server_key_iv: "bcaf2c5587542ddf407afffce1a88c75"
server_key_version: "encryptionkey@0"
I20260925 00:49:15.926849 11079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:15.931249 11079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:15.931803 11079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:15.931977 11079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:15.932479 11079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:49:15.933327 11079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:49:15.933398 11079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.933507 11079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:49:15.933552 11079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:15.941795 11079 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:39551
I20260925 00:49:15.941851 11206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:39551 every 8 connection(s)
I20260925 00:49:15.942680 11079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:49:15.947363 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 11079
I20260925 00:49:15.947485 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:49:15.947846 997 external_mini_cluster.cc:1428] Setting key 2b0896fe4fce21d44e722109298722e4
I20260925 00:49:15.948781 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:15.944437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:40697 (local address 127.0.249.126:45839)
0925 00:49:15.944589 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:15.944593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:15.945577 (+ 984us) server_negotiation.cc:407] Connection header received
0925 00:49:15.946972 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:15.946975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:15.947018 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:15.947082 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:15.948283 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:15.948286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:15.948291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:49:15.948308 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:15.948328 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:49:15.948336 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:15.948462 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:15.948604 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 00:49:15.948649 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:49:15.949790 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:15.944744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:45839 (local address 127.0.249.67:40697)
0925 00:49:15.945360 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:15.945460 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:15.946735 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:15.947252 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:15.947258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:15.947345 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:15.948024 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:49:15.948025 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:15.948066 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:49:15.948131 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:49:15.948440 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:49:15.948448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:49:15.948453 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 00:49:15.948630 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 00:49:15.948854 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260925 00:49:15.950788 11207 heartbeater.cc:347] Connected to a master server at 127.0.249.126:45839
I20260925 00:49:15.951015 11207 heartbeater.cc:464] Registering TS with master...
I20260925 00:49:15.951397 11207 heartbeater.cc:511] Master 127.0.249.126:45839 requested a full tablet report, sending...
I20260925 00:49:15.952317 10757 ts_manager.cc:194] Registered new tserver with Master: 380b599493ee4a2ba4417bb7c4fa00ea (127.0.249.67:39551)
I20260925 00:49:15.961813 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 00:49:15.964751 10796 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:49:15.963751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.0.249.126:45839)
0925 00:49:15.963911 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:15.963915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:15.964095 (+ 180us) server_negotiation.cc:407] Connection header received
0925 00:49:15.964249 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:15.964254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:15.964295 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:15.964394 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:15.964628 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34470: BlockingRecv error: recv got EOF from 127.0.0.1:34470 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:49:15.965081 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 10817
I20260925 00:49:15.970675 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 10948
I20260925 00:49:15.976212 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 11079
I20260925 00:49:15.980980 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 10726
2026-09-25T00:49:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (805 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:49:16 dist-test-slave-05m0 krb5kdc[11217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:49:16 dist-test-slave-05m0 krb5kdc[11217](info): set up 2 sockets
Sep 25 00:49:16 dist-test-slave-05m0 krb5kdc[11217](info): commencing operation
krb5kdc: starting...
W20260925 00:49:17.018574 997 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:49:17 dist-test-slave-05m0 krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:49:17.143749 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:41911
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:41911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:49:17.256409 11230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:17.256681 11230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:17.256726 11230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:17.260875 11230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:49:17.260946 11230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:17.260965 11230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:17.260982 11230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:49:17.260998 11230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:49:17.266218 11230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:41911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:41911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:17.267156 11230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:17.268088 11230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:49:17.276398 11230 server_base.cc:1034] running on GCE node
W20260925 00:49:17.276400 11236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:17.276371 11235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:17.276379 11238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:17.277200 11230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:17.278268 11230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:17.279456 11230 hybrid_clock.cc:648] HybridClock initialized: now 1790297357279425 us; error 44 us; skew 500 ppm
Sep 25 00:49:17 dist-test-slave-05m0 krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:49:17.282123 11230 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:49:17.282971 11230 webserver.cc:460] Webserver started at http://127.0.249.126:41181/ using document root <none> and password file <none>
I20260925 00:49:17.283505 11230 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:17.283593 11230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:17.283818 11230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:17.285837 11230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "c338e367ad184b20a0f8f5abd751a8d8"
format_stamp: "Formatted at 2026-09-25 00:49:17 on dist-test-slave-05m0"
server_key: "d322cf9e4589761baa433d046f8b2fd6"
server_key_iv: "b652167a15fba572fb9452f730b64c84"
server_key_version: "encryptionkey@0"
I20260925 00:49:17.286383 11230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "c338e367ad184b20a0f8f5abd751a8d8"
format_stamp: "Formatted at 2026-09-25 00:49:17 on dist-test-slave-05m0"
server_key: "d322cf9e4589761baa433d046f8b2fd6"
server_key_iv: "b652167a15fba572fb9452f730b64c84"
server_key_version: "encryptionkey@0"
I20260925 00:49:17.289923 11230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:49:17.292217 11245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:17.293478 11230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:49:17.293617 11230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "c338e367ad184b20a0f8f5abd751a8d8"
format_stamp: "Formatted at 2026-09-25 00:49:17 on dist-test-slave-05m0"
server_key: "d322cf9e4589761baa433d046f8b2fd6"
server_key_iv: "b652167a15fba572fb9452f730b64c84"
server_key_version: "encryptionkey@0"
I20260925 00:49:17.293722 11230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:17.322073 11230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:17.326735 11230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:17.326979 11230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:17.335180 11230 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:41911
I20260925 00:49:17.335455 11297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:41911 every 8 connection(s)
I20260925 00:49:17.336192 11230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:49:17.338903 11298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:49:17.339883 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 11230
I20260925 00:49:17.339998 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:49:17.340274 997 external_mini_cluster.cc:1428] Setting key f908e5b46fa35c318069172e45a105fc
I20260925 00:49:17.345032 11298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8: Bootstrap starting.
I20260925 00:49:17.347638 11298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8: Neither blocks nor log segments found. Creating new log.
I20260925 00:49:17.348555 11298 log.cc:824] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:49:17 dist-test-slave-05m0 krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:49:17.350906 11298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8: No bootstrap required, opened a new log
I20260925 00:49:17.354540 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:17.341643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.0.249.126:41911)
0925 00:49:17.342030 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:17.342046 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:17.342081 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:49:17.342687 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:17.342703 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:17.343037 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:17.343307 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:17.344181 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.345237 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.346432 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.346949 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.347080 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:17.350002 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:17.350008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:17.350024 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:49:17.350052 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:17.351761 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:17.352230 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:17.352234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:17.352238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:17.352344 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:17.352621 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:17.352623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:17.352624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:17.353001 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:17.353136 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:17.353505 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 00:49:17.353821 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":125,"threads_started":1}
I20260925 00:49:17.354899 11298 raft_consensus.cc:348] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c338e367ad184b20a0f8f5abd751a8d8" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41911 } }
I20260925 00:49:17.355150 11298 raft_consensus.cc:374] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:49:17.355228 11298 raft_consensus.cc:729] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c338e367ad184b20a0f8f5abd751a8d8, State: Initialized, Role: FOLLOWER
I20260925 00:49:17.355692 11298 consensus_queue.cc:260] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c338e367ad184b20a0f8f5abd751a8d8" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41911 } }
I20260925 00:49:17.355837 11298 raft_consensus.cc:388] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:49:17.355906 11298 raft_consensus.cc:482] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:49:17.356009 11298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:49:17.356848 11298 raft_consensus.cc:504] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c338e367ad184b20a0f8f5abd751a8d8" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41911 } }
I20260925 00:49:17.357182 11298 leader_election.cc:302] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c338e367ad184b20a0f8f5abd751a8d8; no voters:
I20260925 00:49:17.357522 11298 leader_election.cc:288] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:49:17.357618 11303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:49:17.357848 11303 raft_consensus.cc:686] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [term 1 LEADER]: Becoming Leader. State: Replica: c338e367ad184b20a0f8f5abd751a8d8, State: Running, Role: LEADER
I20260925 00:49:17.358142 11303 consensus_queue.cc:237] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c338e367ad184b20a0f8f5abd751a8d8" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41911 } }
I20260925 00:49:17.358709 11298 sys_catalog.cc:564] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:49:17.359726 11304 sys_catalog.cc:455] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c338e367ad184b20a0f8f5abd751a8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c338e367ad184b20a0f8f5abd751a8d8" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41911 } } }
I20260925 00:49:17.359715 11305 sys_catalog.cc:455] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c338e367ad184b20a0f8f5abd751a8d8. Latest consensus state: current_term: 1 leader_uuid: "c338e367ad184b20a0f8f5abd751a8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c338e367ad184b20a0f8f5abd751a8d8" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41911 } } }
I20260925 00:49:17.359860 11304 sys_catalog.cc:458] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [sys.catalog]: This master's current role is: LEADER
I20260925 00:49:17.359860 11305 sys_catalog.cc:458] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8 [sys.catalog]: This master's current role is: LEADER
I20260925 00:49:17.360247 11312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:49:17.363363 11312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:49:17.368875 11312 catalog_manager.cc:1347] Generated new cluster ID: f9ff5af4b57c4d28b3f7e9ec09838fc5
I20260925 00:49:17.368945 11312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:49:17.378701 11312 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:49:17.379586 11312 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:49:17.386914 11312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8: Generated new TSK 0
I20260925 00:49:17.387555 11312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:49:17.420305 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:41911
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:49:17.560645 11324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:17.560915 11324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:17.561019 11324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:17.565042 11324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:17.565105 11324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:17.565164 11324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:49:17.570489 11324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:41911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:17.571481 11324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:17.572422 11324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:17.581924 11330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:17.581914 11324 server_base.cc:1034] running on GCE node
W20260925 00:49:17.581904 11329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:17.581895 11332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:17.582530 11324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:17.583107 11324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:17.584259 11324 hybrid_clock.cc:648] HybridClock initialized: now 1790297357584247 us; error 27 us; skew 500 ppm
Sep 25 00:49:17 dist-test-slave-05m0 krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:49:17.586891 11324 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:49:17.587715 11324 webserver.cc:460] Webserver started at http://127.0.249.65:37069/ using document root <none> and password file <none>
I20260925 00:49:17.588199 11324 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:17.588255 11324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:17.588418 11324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:17.590363 11324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "dc17d8d17dac46cf9a6f65c919a09e7c"
format_stamp: "Formatted at 2026-09-25 00:49:17 on dist-test-slave-05m0"
server_key: "a0bb02adfd739c1a7a949d7e2a9db0ef"
server_key_iv: "4e59b936c7c7cc64e7752475e159a321"
server_key_version: "encryptionkey@0"
I20260925 00:49:17.590859 11324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "dc17d8d17dac46cf9a6f65c919a09e7c"
format_stamp: "Formatted at 2026-09-25 00:49:17 on dist-test-slave-05m0"
server_key: "a0bb02adfd739c1a7a949d7e2a9db0ef"
server_key_iv: "4e59b936c7c7cc64e7752475e159a321"
server_key_version: "encryptionkey@0"
I20260925 00:49:17.594439 11324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:49:17.596741 11339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:17.598053 11324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:49:17.598150 11324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "dc17d8d17dac46cf9a6f65c919a09e7c"
format_stamp: "Formatted at 2026-09-25 00:49:17 on dist-test-slave-05m0"
server_key: "a0bb02adfd739c1a7a949d7e2a9db0ef"
server_key_iv: "4e59b936c7c7cc64e7752475e159a321"
server_key_version: "encryptionkey@0"
I20260925 00:49:17.598232 11324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:17.623324 11324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:17.626844 11324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:17.627017 11324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:17.627561 11324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:49:17.628351 11324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:49:17.628403 11324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:17.628443 11324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:49:17.628458 11324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:17.637247 11324 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:43407
I20260925 00:49:17.637262 11452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:43407 every 8 connection(s)
I20260925 00:49:17.638170 11324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:49:17.647226 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 11324
I20260925 00:49:17.647385 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:49:17.647704 997 external_mini_cluster.cc:1428] Setting key 8a912887d759b63050beb75400b79ac5
Sep 25 00:49:17 dist-test-slave-05m0 krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:49:17.652513 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:17.639932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:42029 (local address 127.0.249.126:41911)
0925 00:49:17.640193 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:17.640198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:17.641006 (+ 808us) server_negotiation.cc:407] Connection header received
0925 00:49:17.642271 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:17.642274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:17.642318 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:17.642395 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:17.643449 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.644007 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.645509 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.645750 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.645899 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:17.648498 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:17.648501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:17.648503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:49:17.648526 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:17.650088 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:17.650773 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:17.650775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:17.650776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:17.650846 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:17.651278 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:17.651283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:17.651287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:17.651537 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:17.651644 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:17.652203 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 00:49:17.652338 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260925 00:49:17.653463 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:17.640439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41911 (local address 127.0.249.65:42029)
0925 00:49:17.640859 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:17.640899 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:17.642090 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:17.642569 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:17.642575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:17.642896 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:17.643331 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:17.643344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.644108 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.644111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:17.645265 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:17.645279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.645931 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.645934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:17.646072 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:17.646640 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:17.646656 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:17.648303 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:49:17.650226 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:49:17.650237 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:49:17.650256 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:49:17.650669 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:49:17.650940 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:49:17.650947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:49:17.650953 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:49:17.651162 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:49:17.651657 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:49:17.651662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:49:17.651950 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 00:49:17.652123 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 00:49:17.652400 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260925 00:49:17.654845 11453 heartbeater.cc:347] Connected to a master server at 127.0.249.126:41911
I20260925 00:49:17.655117 11453 heartbeater.cc:464] Registering TS with master...
I20260925 00:49:17.655755 11453 heartbeater.cc:511] Master 127.0.249.126:41911 requested a full tablet report, sending...
I20260925 00:49:17.657596 11262 ts_manager.cc:194] Registered new tserver with Master: dc17d8d17dac46cf9a6f65c919a09e7c (127.0.249.65:43407)
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:49:17.659264 11262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:42029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:49:17.676551 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:41911
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:49:17.783555 11458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:17.783772 11458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:17.783813 11458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:17.787705 11458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:17.787757 11458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:17.787847 11458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:49:17.792909 11458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:41911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:17.793860 11458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:17.794806 11458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:17.804307 11464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:17.804360 11463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:17.804360 11466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:17.804555 11458 server_base.cc:1034] running on GCE node
I20260925 00:49:17.804895 11458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:17.805569 11458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:17.806723 11458 hybrid_clock.cc:648] HybridClock initialized: now 1790297357806712 us; error 24 us; skew 500 ppm
Sep 25 00:49:17 dist-test-slave-05m0 krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:49:17.809262 11458 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:49:17.810112 11458 webserver.cc:460] Webserver started at http://127.0.249.66:39147/ using document root <none> and password file <none>
I20260925 00:49:17.810595 11458 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:17.810662 11458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:17.810814 11458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:17.812638 11458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "bdc4e146e62e4ccba09df4deb5dae5ea"
format_stamp: "Formatted at 2026-09-25 00:49:17 on dist-test-slave-05m0"
server_key: "150f163f737a377e7d67c4cf9dfb8bee"
server_key_iv: "6311a691522d56a82b7a2504a4c87bed"
server_key_version: "encryptionkey@0"
I20260925 00:49:17.813117 11458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "bdc4e146e62e4ccba09df4deb5dae5ea"
format_stamp: "Formatted at 2026-09-25 00:49:17 on dist-test-slave-05m0"
server_key: "150f163f737a377e7d67c4cf9dfb8bee"
server_key_iv: "6311a691522d56a82b7a2504a4c87bed"
server_key_version: "encryptionkey@0"
I20260925 00:49:17.816584 11458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 00:49:17.818902 11473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:17.820046 11458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:49:17.820173 11458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "bdc4e146e62e4ccba09df4deb5dae5ea"
format_stamp: "Formatted at 2026-09-25 00:49:17 on dist-test-slave-05m0"
server_key: "150f163f737a377e7d67c4cf9dfb8bee"
server_key_iv: "6311a691522d56a82b7a2504a4c87bed"
server_key_version: "encryptionkey@0"
I20260925 00:49:17.820277 11458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:17.861470 11458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:17.864580 11458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:17.864773 11458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:17.865340 11458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:49:17.866140 11458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:49:17.866209 11458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:17.866277 11458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:49:17.866328 11458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:17.874969 11458 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:44803
I20260925 00:49:17.874997 11586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:44803 every 8 connection(s)
I20260925 00:49:17.875830 11458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:49:17.883061 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 11458
I20260925 00:49:17.883169 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:49:17.883519 997 external_mini_cluster.cc:1428] Setting key 3f253c1559501d54574deee5b7d1a1c4
Sep 25 00:49:17 dist-test-slave-05m0 krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:49:17.889647 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:17.877605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:47455 (local address 127.0.249.126:41911)
0925 00:49:17.877735 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:17.877739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:17.878528 (+ 789us) server_negotiation.cc:407] Connection header received
0925 00:49:17.879704 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:17.879707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:17.879751 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:17.879823 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:17.880899 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.881461 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.882555 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.882773 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.882845 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:17.885685 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:17.885692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:17.885698 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:49:17.885731 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:17.887460 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:17.888080 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:17.888083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:17.888085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:17.888177 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:17.888523 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:17.888526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:17.888528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:17.888705 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:17.888815 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:17.889327 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 00:49:17.889474 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:49:17.890410 11589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:17.877975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41911 (local address 127.0.249.66:47455)
0925 00:49:17.878373 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:17.878421 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:17.879524 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:17.880003 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:17.880009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:17.880342 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:17.880780 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:17.880793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.881585 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.881588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:17.882456 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:17.882464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:17.882936 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:17.882944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:17.883146 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:17.883744 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:17.883763 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:17.885528 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:49:17.887574 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:49:17.887583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:49:17.887603 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:49:17.887960 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:49:17.888277 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:49:17.888282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:49:17.888284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:49:17.888417 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:49:17.888809 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:49:17.888815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:49:17.889102 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 00:49:17.889284 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 00:49:17.889611 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
I20260925 00:49:17.891731 11587 heartbeater.cc:347] Connected to a master server at 127.0.249.126:41911
I20260925 00:49:17.891966 11587 heartbeater.cc:464] Registering TS with master...
I20260925 00:49:17.892427 11587 heartbeater.cc:511] Master 127.0.249.126:41911 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:49:17.893509 11262 ts_manager.cc:194] Registered new tserver with Master: bdc4e146e62e4ccba09df4deb5dae5ea (127.0.249.66:44803)
I20260925 00:49:17.894073 11262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:47455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:49:17.911619 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:41911
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:49:18.019958 11592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:49:18.020211 11592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:49:18.020253 11592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:49:18.024356 11592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:49:18.024415 11592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:49:18.024471 11592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:49:18.029687 11592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:41911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:49:18.030643 11592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:49:18.031584 11592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:49:18.040704 11598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:49:18.040643 11597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:18.040844 11592 server_base.cc:1034] running on GCE node
W20260925 00:49:18.040643 11600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:49:18.041272 11592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:49:18.041927 11592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:49:18.043110 11592 hybrid_clock.cc:648] HybridClock initialized: now 1790297358043078 us; error 44 us; skew 500 ppm
Sep 25 00:49:18 dist-test-slave-05m0 krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:49:18.045724 11592 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:49:18.046519 11592 webserver.cc:460] Webserver started at http://127.0.249.67:34027/ using document root <none> and password file <none>
I20260925 00:49:18.047062 11592 fs_manager.cc:359] Metadata directory not provided
I20260925 00:49:18.047147 11592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:49:18.047359 11592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:49:18.049280 11592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "8a5904d849c24eb7bbb8fc1c944d7815"
format_stamp: "Formatted at 2026-09-25 00:49:18 on dist-test-slave-05m0"
server_key: "0f82bf794a2f120fb56133be36561728"
server_key_iv: "90de68414debd12d18a3e4bfd7948663"
server_key_version: "encryptionkey@0"
I20260925 00:49:18.049912 11592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "8a5904d849c24eb7bbb8fc1c944d7815"
format_stamp: "Formatted at 2026-09-25 00:49:18 on dist-test-slave-05m0"
server_key: "0f82bf794a2f120fb56133be36561728"
server_key_iv: "90de68414debd12d18a3e4bfd7948663"
server_key_version: "encryptionkey@0"
I20260925 00:49:18.053550 11592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:49:18.055786 11607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:18.056967 11592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:49:18.057093 11592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "8a5904d849c24eb7bbb8fc1c944d7815"
format_stamp: "Formatted at 2026-09-25 00:49:18 on dist-test-slave-05m0"
server_key: "0f82bf794a2f120fb56133be36561728"
server_key_iv: "90de68414debd12d18a3e4bfd7948663"
server_key_version: "encryptionkey@0"
I20260925 00:49:18.057199 11592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:49:18.078670 11592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:49:18.081496 11592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:49:18.081681 11592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:49:18.082235 11592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:49:18.082990 11592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:49:18.083057 11592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:18.083122 11592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:49:18.083170 11592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:49:18.091850 11592 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:34851
I20260925 00:49:18.091892 11720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:34851 every 8 connection(s)
I20260925 00:49:18.092686 11592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:49:18.097561 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 11592
I20260925 00:49:18.097703 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:49:18.098026 997 external_mini_cluster.cc:1428] Setting key 25a89553600538259f4b19941c7c3d02
Sep 25 00:49:18 dist-test-slave-05m0 krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:49:18.107641 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.094387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:36477 (local address 127.0.249.126:41911)
0925 00:49:18.094529 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:18.094533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:18.095311 (+ 778us) server_negotiation.cc:407] Connection header received
0925 00:49:18.096742 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:18.096746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:18.096804 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:18.096887 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:18.098310 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.099057 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.100416 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.100649 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.100758 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.103731 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:18.103735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:18.103737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:49:18.103784 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:18.105637 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:18.106136 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:18.106141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:18.106143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:18.106225 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:18.106592 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:18.106595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:18.106597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:18.106772 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:18.106877 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:18.107331 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 00:49:18.107444 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 00:49:18.108340 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.094758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:41911 (local address 127.0.249.67:36477)
0925 00:49:18.095167 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:18.095207 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:18.096532 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:18.097090 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:18.097097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:18.097560 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:49:18.098170 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.098191 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.099176 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.099179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.100295 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.100304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.100790 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.100794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.100975 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.101569 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:49:18.101590 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:49:18.103521 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:49:18.105761 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:49:18.105766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:49:18.105779 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:49:18.106034 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:49:18.106332 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:49:18.106335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:49:18.106355 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:49:18.106492 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:49:18.106874 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:49:18.106880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:49:18.107110 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 00:49:18.107290 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 00:49:18.107587 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
I20260925 00:49:18.109460 11721 heartbeater.cc:347] Connected to a master server at 127.0.249.126:41911
I20260925 00:49:18.109689 11721 heartbeater.cc:464] Registering TS with master...
I20260925 00:49:18.110133 11721 heartbeater.cc:511] Master 127.0.249.126:41911 requested a full tablet report, sending...
I20260925 00:49:18.111229 11262 ts_manager.cc:194] Registered new tserver with Master: 8a5904d849c24eb7bbb8fc1c944d7815 (127.0.249.67:34851)
I20260925 00:49:18.111732 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:49:18.111900 11262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:36477
I20260925 00:49:18.120771 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.113802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.0.249.126:41911)
0925 00:49:18.113909 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:18.113915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:18.114106 (+ 191us) server_negotiation.cc:407] Connection header received
0925 00:49:18.114234 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:18.114237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:18.114274 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:18.114363 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:49:18.115039 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.115547 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.116368 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.116602 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.116723 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.118031 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:49:18.118035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:49:18.118037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:49:18.118060 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:49:18.119189 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:18.119593 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:18.119598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:18.119600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:18.119680 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:49:18.119955 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:49:18.119959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:49:18.119961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:49:18.120161 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:49:18.120298 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:18.120522 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 00:49:18.120626 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 00:49:18.123234 11262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:49:18.126277 11262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:49:18.139914 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.134061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.0.249.65:43407)
0925 00:49:18.134722 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:18.134741 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:18.134768 (+ 27us) server_negotiation.cc:407] Connection header received
0925 00:49:18.134847 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:18.134853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:18.135056 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:18.135237 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:49:18.135541 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.136681 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.138342 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.139179 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.139317 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.139456 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:18.139661 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 00:49:18.139768 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":100,"threads_started":1}
I20260925 00:49:18.139933 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.134057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:44803 (local address 127.0.0.1:45182)
0925 00:49:18.134958 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:18.134972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:18.135067 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:18.135981 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:18.135984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:18.136008 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:49:18.136103 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.136111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.137332 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.137338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.138334 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.138343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.139537 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.139540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.139638 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.139668 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 00:49:18.139739 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 00:49:18.139799 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":4,"thread_start_us":92,"threads_started":1}
I20260925 00:49:18.140213 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.134866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.0.249.67:34851)
0925 00:49:18.135715 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:18.135734 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:18.135768 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:49:18.135837 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:18.135842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:18.136025 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:18.136153 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:49:18.136363 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.137230 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.138677 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.139346 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.139445 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.139575 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:18.139963 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 00:49:18.140076 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":185,"threads_started":1}
I20260925 00:49:18.140422 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.134686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:34851 (local address 127.0.0.1:49300)
0925 00:49:18.135618 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:18.135629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:18.135704 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:18.136162 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:18.136165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:18.136184 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:49:18.136257 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.136266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.137382 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.137386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.138550 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.138561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.139636 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.139640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.139720 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.139739 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 00:49:18.139804 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 00:49:18.139863 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":21,"thread_start_us":87,"threads_started":1}
I20260925 00:49:18.139902 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.133954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:43407 (local address 127.0.0.1:48332)
0925 00:49:18.134283 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:18.134326 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:18.134440 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:18.135221 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:18.135229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:18.135253 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:49:18.135389 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.135396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.136817 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.136820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.138218 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.138228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.139330 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.139334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.139449 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.139485 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 00:49:18.139625 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 00:49:18.139736 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":110,"threads_started":1}
I20260925 00:49:18.140775 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.135066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.0.249.66:44803)
0925 00:49:18.135549 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:18.135562 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:18.135584 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:49:18.135665 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:18.135673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:18.135853 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:18.135999 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:49:18.136229 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.137160 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.138481 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.139225 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.139318 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.139455 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:18.140282 (+ 827us) server_negotiation.cc:299] Negotiation successful
0925 00:49:18.140400 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":120,"threads_started":1}
I20260925 00:49:18.143009 11655 tablet_service.cc:1467] Processing CreateTablet for tablet a36da79dde2644d18246eac2d6e4abfc (DEFAULT_TABLE table=test-table [id=76b4f78c4e9b4e2ea4f7ddfaf8345509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:49:18.143004 11387 tablet_service.cc:1467] Processing CreateTablet for tablet a36da79dde2644d18246eac2d6e4abfc (DEFAULT_TABLE table=test-table [id=76b4f78c4e9b4e2ea4f7ddfaf8345509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:49:18.143864 11387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36da79dde2644d18246eac2d6e4abfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:49:18.143873 11655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36da79dde2644d18246eac2d6e4abfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:49:18.147374 11521 tablet_service.cc:1467] Processing CreateTablet for tablet a36da79dde2644d18246eac2d6e4abfc (DEFAULT_TABLE table=test-table [id=76b4f78c4e9b4e2ea4f7ddfaf8345509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:49:18.148391 11521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36da79dde2644d18246eac2d6e4abfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:49:18.149521 11738 tablet_bootstrap.cc:492] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c: Bootstrap starting.
I20260925 00:49:18.151084 11739 tablet_bootstrap.cc:492] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: Bootstrap starting.
I20260925 00:49:18.151926 11738 tablet_bootstrap.cc:654] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c: Neither blocks nor log segments found. Creating new log.
I20260925 00:49:18.152927 11738 log.cc:824] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c: Log is configured to *not* fsync() on all Append() calls
I20260925 00:49:18.153558 11739 tablet_bootstrap.cc:654] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: Neither blocks nor log segments found. Creating new log.
I20260925 00:49:18.154392 11739 log.cc:824] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: Log is configured to *not* fsync() on all Append() calls
I20260925 00:49:18.154875 11738 tablet_bootstrap.cc:492] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c: No bootstrap required, opened a new log
I20260925 00:49:18.155081 11738 ts_tablet_manager.cc:1397] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:49:18.155884 11742 tablet_bootstrap.cc:492] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: Bootstrap starting.
I20260925 00:49:18.156309 11739 tablet_bootstrap.cc:492] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: No bootstrap required, opened a new log
I20260925 00:49:18.156471 11739 ts_tablet_manager.cc:1397] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:49:18.158277 11742 tablet_bootstrap.cc:654] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: Neither blocks nor log segments found. Creating new log.
I20260925 00:49:18.158181 11738 raft_consensus.cc:348] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } }
I20260925 00:49:18.158428 11738 raft_consensus.cc:374] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:49:18.158496 11738 raft_consensus.cc:729] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc17d8d17dac46cf9a6f65c919a09e7c, State: Initialized, Role: FOLLOWER
I20260925 00:49:18.158921 11738 consensus_queue.cc:260] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } }
I20260925 00:49:18.159085 11742 log.cc:824] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: Log is configured to *not* fsync() on all Append() calls
I20260925 00:49:18.159590 11453 heartbeater.cc:503] Master 127.0.249.126:41911 was elected leader, sending a full tablet report...
I20260925 00:49:18.160005 11738 ts_tablet_manager.cc:1428] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260925 00:49:18.160563 11739 raft_consensus.cc:348] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } }
I20260925 00:49:18.160972 11742 tablet_bootstrap.cc:492] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: No bootstrap required, opened a new log
I20260925 00:49:18.160948 11739 raft_consensus.cc:374] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:49:18.161024 11739 raft_consensus.cc:729] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5904d849c24eb7bbb8fc1c944d7815, State: Initialized, Role: FOLLOWER
I20260925 00:49:18.161144 11742 ts_tablet_manager.cc:1397] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:49:18.161505 11739 consensus_queue.cc:260] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } }
I20260925 00:49:18.162432 11739 ts_tablet_manager.cc:1428] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:49:18.163098 11721 heartbeater.cc:503] Master 127.0.249.126:41911 was elected leader, sending a full tablet report...
I20260925 00:49:18.164445 11742 raft_consensus.cc:348] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } }
I20260925 00:49:18.164696 11742 raft_consensus.cc:374] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:49:18.164757 11742 raft_consensus.cc:729] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc4e146e62e4ccba09df4deb5dae5ea, State: Initialized, Role: FOLLOWER
I20260925 00:49:18.165102 11742 consensus_queue.cc:260] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } }
I20260925 00:49:18.165709 11587 heartbeater.cc:503] Master 127.0.249.126:41911 was elected leader, sending a full tablet report...
I20260925 00:49:18.166007 11742 ts_tablet_manager.cc:1428] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 00:49:18.343566 11722 tablet.cc:2369] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:49:18.357064 11745 raft_consensus.cc:482] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:49:18.357321 11745 raft_consensus.cc:504] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } }
I20260925 00:49:18.358639 11745 leader_election.cc:288] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc17d8d17dac46cf9a6f65c919a09e7c (127.0.249.65:43407), bdc4e146e62e4ccba09df4deb5dae5ea (127.0.249.66:44803)
I20260925 00:49:18.363055 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.359086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:51451 (local address 127.0.249.66:44803)
0925 00:49:18.359226 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:18.359230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:18.359323 (+ 93us) server_negotiation.cc:407] Connection header received
0925 00:49:18.359468 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:18.359472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:18.359539 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:18.359620 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:49:18.359950 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.360639 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.362094 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.362477 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.362599 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.362656 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:18.362864 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 00:49:18.362929 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:49:18.363078 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.359050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:43407 (local address 127.0.249.67:54453)
0925 00:49:18.359454 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:18.359468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:18.359595 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:18.359934 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:18.359937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:18.359956 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:49:18.360059 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.360065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.360940 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.360943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.361918 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.361927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.362649 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.362652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.362748 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.362764 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 00:49:18.362852 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 00:49:18.362925 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":42,"thread_start_us":121,"threads_started":1}
I20260925 00:49:18.363081 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.359158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:54453 (local address 127.0.249.65:43407)
0925 00:49:18.359320 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:18.359325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:18.359569 (+ 244us) server_negotiation.cc:407] Connection header received
0925 00:49:18.359705 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:18.359709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:18.359761 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:18.359861 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:49:18.360158 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.360819 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.362056 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.362498 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.362645 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.362705 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:18.362891 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 00:49:18.362959 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 00:49:18.363081 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.358999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:44803 (local address 127.0.249.67:51451)
0925 00:49:18.359201 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:18.359219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:49:18.359338 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:49:18.359668 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:49:18.359672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:49:18.359694 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:49:18.359844 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.359855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.360757 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.360760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.361930 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:49:18.361940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.362605 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.362610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:49:18.362715 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.362747 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 00:49:18.362852 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 00:49:18.362937 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":42}
I20260925 00:49:18.363860 11407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36da79dde2644d18246eac2d6e4abfc" candidate_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" is_pre_election: true
I20260925 00:49:18.364013 11541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36da79dde2644d18246eac2d6e4abfc" candidate_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" is_pre_election: true
I20260925 00:49:18.364193 11407 raft_consensus.cc:2454] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5904d849c24eb7bbb8fc1c944d7815 in term 0.
I20260925 00:49:18.364331 11541 raft_consensus.cc:2454] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5904d849c24eb7bbb8fc1c944d7815 in term 0.
I20260925 00:49:18.364728 11610 leader_election.cc:302] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5904d849c24eb7bbb8fc1c944d7815, dc17d8d17dac46cf9a6f65c919a09e7c; no voters:
I20260925 00:49:18.365000 11745 raft_consensus.cc:2781] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:49:18.365084 11745 raft_consensus.cc:482] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:49:18.365142 11745 raft_consensus.cc:3037] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:49:18.366250 11745 raft_consensus.cc:504] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } }
I20260925 00:49:18.366627 11745 leader_election.cc:288] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [CANDIDATE]: Term 1 election: Requested vote from peers dc17d8d17dac46cf9a6f65c919a09e7c (127.0.249.65:43407), bdc4e146e62e4ccba09df4deb5dae5ea (127.0.249.66:44803)
I20260925 00:49:18.367063 11541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36da79dde2644d18246eac2d6e4abfc" candidate_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea"
I20260925 00:49:18.367210 11541 raft_consensus.cc:3037] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:49:18.367218 11407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36da79dde2644d18246eac2d6e4abfc" candidate_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c"
I20260925 00:49:18.367375 11407 raft_consensus.cc:3037] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:49:18.368270 11541 raft_consensus.cc:2454] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5904d849c24eb7bbb8fc1c944d7815 in term 1.
I20260925 00:49:18.368407 11407 raft_consensus.cc:2454] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5904d849c24eb7bbb8fc1c944d7815 in term 1.
I20260925 00:49:18.368572 11611 leader_election.cc:302] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5904d849c24eb7bbb8fc1c944d7815, bdc4e146e62e4ccba09df4deb5dae5ea; no voters:
I20260925 00:49:18.368780 11745 raft_consensus.cc:2781] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:49:18.368966 11745 raft_consensus.cc:686] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 1 LEADER]: Becoming Leader. State: Replica: 8a5904d849c24eb7bbb8fc1c944d7815, State: Running, Role: LEADER
I20260925 00:49:18.369277 11745 consensus_queue.cc:237] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } }
I20260925 00:49:18.372488 11262 catalog_manager.cc:5495] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a5904d849c24eb7bbb8fc1c944d7815 (127.0.249.67). New cstate: current_term: 1 leader_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5904d849c24eb7bbb8fc1c944d7815" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 34851 } health_report { overall_health: HEALTHY } } }
W20260925 00:49:18.376890 11588 tablet.cc:2369] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:49:18.389433 11454 tablet.cc:2369] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:49:18.436172 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.432926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.0.249.67:34851)
0925 00:49:18.433060 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:49:18.433063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:49:18.433077 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:49:18.433187 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:49:18.433190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:49:18.433242 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:49:18.433330 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:49:18.433593 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.434072 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.434867 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:49:18.435066 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:49:18.435142 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:49:18.435386 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:49:18.435791 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:49:18.435877 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:49:18.435985 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 00:49:18.436034 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:49:18.441939 11407 raft_consensus.cc:1261] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [term 1 FOLLOWER]: Refusing update from remote peer 8a5904d849c24eb7bbb8fc1c944d7815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:49:18.441941 11541 raft_consensus.cc:1261] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [term 1 FOLLOWER]: Refusing update from remote peer 8a5904d849c24eb7bbb8fc1c944d7815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:49:18.442648 11748 consensus_queue.cc:1035] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc17d8d17dac46cf9a6f65c919a09e7c" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:49:18.442816 11745 consensus_queue.cc:1035] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdc4e146e62e4ccba09df4deb5dae5ea" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:49:18.450256 11753 mvcc.cc:205] Tried to move back new op lower bound from 7333057980171210752 to 7333057979883364352. Current Snapshot: MvccSnapshot[applied={T|T < 7333057980171210752}]
I20260925 00:49:18.450367 11751 mvcc.cc:205] Tried to move back new op lower bound from 7333057980171210752 to 7333057979883364352. Current Snapshot: MvccSnapshot[applied={T|T < 7333057980171210752}]
I20260925 00:49:18.460624 11260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:49:18.460867 11260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:49:18.463340 11260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c338e367ad184b20a0f8f5abd751a8d8: Sending DeleteTablet for 3 replicas of tablet a36da79dde2644d18246eac2d6e4abfc
I20260925 00:49:18.464107 11387 tablet_service.cc:1514] Processing DeleteTablet for tablet a36da79dde2644d18246eac2d6e4abfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:49:18 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:48332
I20260925 00:49:18.464229 11521 tablet_service.cc:1514] Processing DeleteTablet for tablet a36da79dde2644d18246eac2d6e4abfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:49:18 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:45182
I20260925 00:49:18.464754 11761 tablet_replica.cc:330] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c: stopping tablet replica
I20260925 00:49:18.464753 11762 tablet_replica.cc:330] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: stopping tablet replica
I20260925 00:49:18.465044 11655 tablet_service.cc:1514] Processing DeleteTablet for tablet a36da79dde2644d18246eac2d6e4abfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:49:18 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:49300
I20260925 00:49:18.465087 11762 raft_consensus.cc:2229] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:49:18.465087 11761 raft_consensus.cc:2229] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:49:18.465451 11762 raft_consensus.cc:2258] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:49:18.465451 11761 raft_consensus.cc:2258] T a36da79dde2644d18246eac2d6e4abfc P dc17d8d17dac46cf9a6f65c919a09e7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:49:18.465637 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 11324
I20260925 00:49:18.466297 11763 tablet_replica.cc:330] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: stopping tablet replica
I20260925 00:49:18.466583 11763 raft_consensus.cc:2229] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:49:18.466959 11763 raft_consensus.cc:2258] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:49:18.467535 11762 ts_tablet_manager.cc:1905] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:49:18.469823 11762 ts_tablet_manager.cc:1918] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:49:18.469924 11762 log.cc:1196] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/a36da79dde2644d18246eac2d6e4abfc
I20260925 00:49:18.470256 11762 ts_tablet_manager.cc:1939] T a36da79dde2644d18246eac2d6e4abfc P bdc4e146e62e4ccba09df4deb5dae5ea: Deleting consensus metadata
I20260925 00:49:18.471505 11249 catalog_manager.cc:4841] TS bdc4e146e62e4ccba09df4deb5dae5ea (127.0.249.66:44803): tablet a36da79dde2644d18246eac2d6e4abfc (table test-table [id=76b4f78c4e9b4e2ea4f7ddfaf8345509]) successfully deleted
I20260925 00:49:18.472546 11763 ts_tablet_manager.cc:1905] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:49:18.475209 11248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:43407 (error 108)
I20260925 00:49:18.475741 11763 ts_tablet_manager.cc:1918] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:49:18.475843 11763 log.cc:1196] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/a36da79dde2644d18246eac2d6e4abfc
I20260925 00:49:18.476195 11763 ts_tablet_manager.cc:1939] T a36da79dde2644d18246eac2d6e4abfc P 8a5904d849c24eb7bbb8fc1c944d7815: Deleting consensus metadata
I20260925 00:49:18.476207 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:49:18.475932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:43407 (local address 127.0.0.1:48348)
0925 00:49:18.476051 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 00:49:18.476122 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 00:49:18.476307 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 11458
W20260925 00:49:18.476454 11248 catalog_manager.cc:4584] TS dc17d8d17dac46cf9a6f65c919a09e7c (127.0.249.65:43407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a36da79dde2644d18246eac2d6e4abfc: Network error: Client connection negotiation failed: client connection to 127.0.249.65:43407: connect: Connection refused (error 111)
I20260925 00:49:18.477288 11249 catalog_manager.cc:4841] TS 8a5904d849c24eb7bbb8fc1c944d7815 (127.0.249.67:34851): tablet a36da79dde2644d18246eac2d6e4abfc (table test-table [id=76b4f78c4e9b4e2ea4f7ddfaf8345509]) successfully deleted
I20260925 00:49:18.482398 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 11592
I20260925 00:49:18.487665 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 11230
2026-09-25T00:49:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2507 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:49:18 dist-test-slave-05m0 krb5kdc[11767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:49:18 dist-test-slave-05m0 krb5kdc[11767](info): set up 2 sockets
Sep 25 00:49:18 dist-test-slave-05m0 krb5kdc[11767](info): commencing operation
krb5kdc: starting...
W20260925 00:49:19.526546 997 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:49:19 dist-test-slave-05m0 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:49:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.249.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.249.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/rangeradmin_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/rangeradmin_127.0.249.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.249.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.249.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/rangerlookup_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/rangerlookup_127.0.249.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.249.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.249.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/HTTP_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/HTTP_127.0.249.84@KRBTEST.COM.keytab.
I20260925 00:49:19.689541 997 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyp5H6g/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/postgres -l logfile start
2026-09-25 00:49:22.137 UTC [11797] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 00:49:22.137 UTC [11797] LOG: listening on IPv4 address "127.0.249.84", port 55477
2026-09-25 00:49:22.140 UTC [11797] LOG: listening on Unix socket "/tmp/.s.PGSQL.55477"
2026-09-25 00:49:22.174 UTC [11800] LOG: database system was shut down at 2026-09-25 00:49:20 UTC
2026-09-25 00:49:22.178 UTC [11797] LOG: database system is ready to accept connections
I20260925 00:49:23.218417 997 mini_postgres.cc:96] Postgres bound to 55477
2026-09-25 00:49:23.224 UTC [11808] FATAL: database "slave" does not exist
127.0.249.84:55477 - accepting connections
I20260925 00:49:23.225283 997 mini_ranger.cc:161] Starting Ranger...
I20260925 00:49:23.240082 997 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 00:49:23.694142 997 mini_ranger.cc:87] Created ranger Postgres database
I20260925 00:49:23.694255 997 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-admin
2026-09-25 00:49:23,996 [I] DB FLAVOR :POSTGRES
2026-09-25 00:49:23,996 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 00:49:23,996 [I] Checking connection..
2026-09-25 00:49:23,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 00:49:24,223 [I] Checking connection passed.
2026-09-25 00:49:24,224 [I] --------- Verifying version history table ---------
2026-09-25 00:49:24,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:49:24,460 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 00:49:24,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:49:24,716 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 00:49:24,717 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 00:49:24,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 00:49:24.958 UTC [11872] WARNING: there is no transaction in progress
2026-09-25 00:49:24,964 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 00:49:24,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:49:25,204 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 00:49:25,204 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 00:49:25,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 00:49:25,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 00:49:25,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-05m0.c.gcp-upstream.internal','N') ;"
2026-09-25 00:49:25,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 00:49:26,148 [I] Table x_portal_user does not exist in database ranger
2026-09-25 00:49:26,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 00:49:26,402 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 00:49:26,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 00:49:26,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 00:49:26,865 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 00:49:26,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 00:49:27.801 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.817 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.832 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.846 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.861 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.879 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.941 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.948 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.962 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.972 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.981 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:27.995 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:28.004 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:28.011 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:28.017 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:28.024 UTC [11980] WARNING: there is no transaction in progress
2026-09-25 00:49:28,648 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 00:49:28,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 00:49:28,874 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 00:49:28,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 00:49:29,106 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 00:49:29,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 00:49:29,334 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 00:49:29,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 00:49:29,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 00:49:29,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-05m0.c.gcp-upstream.internal';"
2026-09-25 00:49:30,022 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 00:49:30,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 00:49:30,272 [I] DB_PATCHES have already been applied
I20260925 00:49:30.277879 997 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyp5H6g/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 00:49:30.277943 997 mini_ranger.cc:193] Using host: 127.0.249.84
I20260925 00:49:30.281358 997 mini_ranger.cc:227] Ranger admin URL: http://127.0.249.84:37011
Sep 25, 2026 12:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 12:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 12:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 12:49:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 12:49:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 00:49:31.391798 997 mini_ranger.cc:160] Time spent starting Ranger: real 8.167s user 0.000s sys 0.010s
Sep 25, 2026 12:49:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 12:49:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 25, 2026 12:49:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 12:49:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 12:49:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 00:49:53.233471 997 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.249.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.249.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/rangerkms_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/rangerkms_127.0.249.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.249.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.249.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.249.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/HTTP_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.249.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/HTTP_127.0.249.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 00:49:53 dist-test-slave-05m0 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297393, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 00:49:53.315270 997 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 00:49:53.326454 997 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 00:49:54.155571 997 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 00:49:54.155669 997 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-kms
I20260925 00:49:54.155773 997 mini_ranger_kms.cc:227] Using postgres at 127.0.249.84:55477
2026-09-25 00:49:54,334 [I] DB FLAVOR :POSTGRES
2026-09-25 00:49:54,334 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 00:49:54,334 [I] Checking connection
2026-09-25 00:49:54,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 00:49:54,599 [I] connection success
2026-09-25 00:49:54,599 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 00:49:54,599 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 00:49:54,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 00:49:54,826 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 00:49:54,826 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 00:49:54,826 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 00:49:54,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:55477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 00:49:55,103 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 00:49:55.596350 997 mini_ranger_kms.cc:310] Created kms service
I20260925 00:49:55.779858 997 mini_ranger_kms.cc:326] Created kudu user
I20260925 00:49:55.824358 997 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 00:49:55 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297393, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:49:56.208535 997 mini_ranger_kms.cc:384] Added ranger policy
I20260925 00:49:56.208698 997 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 00:49:56.208720 997 mini_ranger_kms.cc:240] Using host: 127.0.249.84
I20260925 00:49:56.212708 997 mini_ranger_kms.cc:276] Ranger KMS PID: 12160
I20260925 00:49:56.212859 997 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.249.84:56121
Sep 25, 2026 12:49:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:49:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:49:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 12:49:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 12:49:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 12:49:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 00:49:58 dist-test-slave-05m0 krb5kdc[11767](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297398, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.249.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:49:58.494230 997 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.179s user 0.007s sys 0.157s
I20260925 00:49:58.494364 997 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 00:49:58.494473 997 mini_ranger_kms.cc:401] 127.0.249.84:56121/kms/v1/keys
Sep 25 00:49:58 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297398, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.249.84@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
Sep 25 00:49:59 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297398, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.249.84@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
Sep 25 00:50:01 dist-test-slave-05m0 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:50:01.411005 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:33691
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:33691
--ranger_config_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:56121/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:01.542261 12211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:01.542557 12211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:01.542603 12211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:01.546887 12211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:01.546968 12211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:01.546991 12211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:01.547011 12211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:01.547070 12211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:01.552855 12211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.249.84:56121/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:33691
--ranger_config_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:01.554085 12211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:01.555166 12211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:01.565380 12219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:01.565497 12211 server_base.cc:1034] running on GCE node
W20260925 00:50:01.565434 12217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:01.565384 12216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:01.566465 12211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:01.567834 12211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:01.569025 12211 hybrid_clock.cc:648] HybridClock initialized: now 1790297401569003 us; error 54 us; skew 500 ppm
Sep 25 00:50:01 dist-test-slave-05m0 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:01.576130 12211 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:50:01.577230 12211 webserver.cc:460] Webserver started at http://127.0.249.126:38075/ using document root <none> and password file <none>
I20260925 00:50:01.577917 12211 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:01.577980 12211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:01.578219 12211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:50:01 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:01.629201 12211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "bd2aaaf1f14c4ace83401db439ff7f67"
format_stamp: "Formatted at 2026-09-25 00:50:01 on dist-test-slave-05m0"
server_key: "37e5f85d2868ccc34de21b7b255dc7be"
server_key_iv: "170a80ffae612113a98cc2c9680b5b6c"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:01.629964 12211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "bd2aaaf1f14c4ace83401db439ff7f67"
format_stamp: "Formatted at 2026-09-25 00:50:01 on dist-test-slave-05m0"
server_key: "37e5f85d2868ccc34de21b7b255dc7be"
server_key_iv: "170a80ffae612113a98cc2c9680b5b6c"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:01.634047 12211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:50:01.648057 12226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:01.649653 12211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:50:01.649789 12211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "bd2aaaf1f14c4ace83401db439ff7f67"
format_stamp: "Formatted at 2026-09-25 00:50:01 on dist-test-slave-05m0"
server_key: "37e5f85d2868ccc34de21b7b255dc7be"
server_key_iv: "170a80ffae612113a98cc2c9680b5b6c"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:01.649928 12211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:01.667073 12211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:01.670236 12211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:01.670456 12211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:01.678858 12211 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:33691
I20260925 00:50:01.678893 12278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:33691 every 8 connection(s)
I20260925 00:50:01.679863 12211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:50:01.682654 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:01.688012 12279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67: Bootstrap starting.
I20260925 00:50:01.689838 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 12211
I20260925 00:50:01.689961 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:50:01.690222 997 external_mini_cluster.cc:1428] Setting key 1dcfd2770242e6e967c831510f77ed94
I20260925 00:50:01.690481 12279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:01.691447 12279 log.cc:824] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:01.693795 12279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67: No bootstrap required, opened a new log
I20260925 00:50:01.698427 12279 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2aaaf1f14c4ace83401db439ff7f67" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33691 } }
I20260925 00:50:01.698740 12279 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:01.698814 12279 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2aaaf1f14c4ace83401db439ff7f67, State: Initialized, Role: FOLLOWER
I20260925 00:50:01.699606 12279 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2aaaf1f14c4ace83401db439ff7f67" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33691 } }
I20260925 00:50:01.699770 12279 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:50:01.699858 12279 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:50:01.699988 12279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 00:50:01 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:01.701236 12279 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2aaaf1f14c4ace83401db439ff7f67" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33691 } }
I20260925 00:50:01.701627 12279 leader_election.cc:302] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd2aaaf1f14c4ace83401db439ff7f67; no voters:
I20260925 00:50:01.702373 12279 leader_election.cc:288] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:50:01.702520 12284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:50:01.702724 12284 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [term 1 LEADER]: Becoming Leader. State: Replica: bd2aaaf1f14c4ace83401db439ff7f67, State: Running, Role: LEADER
I20260925 00:50:01.703075 12284 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2aaaf1f14c4ace83401db439ff7f67" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33691 } }
I20260925 00:50:01.704080 12279 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:50:01.704662 12284 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd2aaaf1f14c4ace83401db439ff7f67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2aaaf1f14c4ace83401db439ff7f67" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33691 } } }
I20260925 00:50:01.705492 12284 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:01.705360 12285 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd2aaaf1f14c4ace83401db439ff7f67. Latest consensus state: current_term: 1 leader_uuid: "bd2aaaf1f14c4ace83401db439ff7f67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2aaaf1f14c4ace83401db439ff7f67" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 33691 } } }
I20260925 00:50:01.705878 12285 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:01.706163 12293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:50:01.706450 12279 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 00:50:01.707706 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:01.691404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.0.249.126:33691)
0925 00:50:01.691908 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:01.691927 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:01.691976 (+ 49us) server_negotiation.cc:407] Connection header received
0925 00:50:01.692791 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:01.692814 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:01.693206 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:01.693610 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:01.694443 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:01.696016 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:01.697638 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:01.698324 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:01.698538 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:01.701490 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:01.701496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:01.701513 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:01.701542 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:01.703621 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:01.704125 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:01.704132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:01.704142 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:01.704283 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:01.704640 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:01.704644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:01.704645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:01.705232 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:01.705456 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:01.706420 (+ 964us) server_negotiation.cc:299] Negotiation successful
0925 00:50:01.706709 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":116,"threads_started":1}
I20260925 00:50:01.709336 12293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:50:01.719898 12293 catalog_manager.cc:1347] Generated new cluster ID: 6320a7951fb2462a9b27349ec19e0121
I20260925 00:50:01.719986 12293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:50:01.743736 12293 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:50:01.745194 12293 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:50:01.751622 12293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67: Generated new TSK 0
I20260925 00:50:01.752620 12293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:50:02.532652 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:56121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33691
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:02.699187 12324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:02.699483 12324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:02.699564 12324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:02.704275 12324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:02.704360 12324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:02.704455 12324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:50:02.710876 12324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.249.84:56121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:02.712392 12324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:02.713685 12324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:02.727433 12333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:02.728030 12331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:02.728087 12330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:02.729696 12324 server_base.cc:1034] running on GCE node
I20260925 00:50:02.730216 12324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:02.731016 12324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:02.732223 12324 hybrid_clock.cc:648] HybridClock initialized: now 1790297402732193 us; error 51 us; skew 500 ppm
Sep 25 00:50:02 dist-test-slave-05m0 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:02.735420 12324 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:50:02.736609 12324 webserver.cc:460] Webserver started at http://127.0.249.65:43943/ using document root <none> and password file <none>
I20260925 00:50:02.737355 12324 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:02.737515 12324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:02.737766 12324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:50:02 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:02.755475 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "0a1a977691904c9bb48e4eab5ed29727"
format_stamp: "Formatted at 2026-09-25 00:50:02 on dist-test-slave-05m0"
server_key: "d62a4eda40b2c703d3931dd8fbd04d67"
server_key_iv: "afd8b1b6dd14fb8ca36a05b430f8f580"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:02.756271 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "0a1a977691904c9bb48e4eab5ed29727"
format_stamp: "Formatted at 2026-09-25 00:50:02 on dist-test-slave-05m0"
server_key: "d62a4eda40b2c703d3931dd8fbd04d67"
server_key_iv: "afd8b1b6dd14fb8ca36a05b430f8f580"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:02.771607 12324 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.002s sys 0.004s
I20260925 00:50:02.789695 12341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:02.791435 12324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:50:02.791569 12324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "0a1a977691904c9bb48e4eab5ed29727"
format_stamp: "Formatted at 2026-09-25 00:50:02 on dist-test-slave-05m0"
server_key: "d62a4eda40b2c703d3931dd8fbd04d67"
server_key_iv: "afd8b1b6dd14fb8ca36a05b430f8f580"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:02.791705 12324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:02.836764 12324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:02.841454 12324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:02.841763 12324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:02.842592 12324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:02.843900 12324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:02.844100 12324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:02.844264 12324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:02.844372 12324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:02.858232 12324 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:39031
I20260925 00:50:02.859560 12324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:50:02.865545 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 12324
I20260925 00:50:02.865682 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:50:02.866000 997 external_mini_cluster.cc:1428] Setting key fc0064f06a98ed29f9b937f2d1fa674d
I20260925 00:50:02.861627 12456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:39031 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
Sep 25 00:50:02 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
I20260925 00:50:02.891906 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:02.864128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:33691 (local address 127.0.249.65:56819)
0925 00:50:02.870783 (+ 6655us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:02.870845 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:02.871943 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:02.876067 (+ 4124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:02.876079 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:02.876479 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:02.877240 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:02.877263 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:02.878560 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:02.878565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:02.880060 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:02.880073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:02.880909 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:02.880914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:02.881126 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:02.882233 (+ 1107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:02.882256 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:02.883883 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:02.886435 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:02.886443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:02.886462 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:02.888530 (+ 2068us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:02.888944 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:02.888949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:02.888951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:02.889167 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:02.889701 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:02.889708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:02.890076 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 00:50:02.890301 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 00:50:02.890668 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6391,"thread_start_us":150,"threads_started":1}
I20260925 00:50:02.892366 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:02.866613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:56819 (local address 127.0.249.126:33691)
0925 00:50:02.872417 (+ 5804us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:02.872423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:02.872440 (+ 17us) server_negotiation.cc:407] Connection header received
0925 00:50:02.872488 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:02.872491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:02.872546 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:02.872652 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:02.877493 (+ 4841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:02.878384 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:02.880228 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:02.880592 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:02.880704 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:02.884103 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:02.884109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:02.884112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:02.884145 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:02.886276 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:02.888691 (+ 2415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:02.888695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:02.888697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:02.888796 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:02.889301 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:02.889306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:02.889308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:02.889572 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:02.891931 (+ 2359us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:02.892043 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 00:50:02.892188 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5712,"thread_start_us":106,"threads_started":1}
I20260925 00:50:02.893738 12457 heartbeater.cc:347] Connected to a master server at 127.0.249.126:33691
I20260925 00:50:02.894023 12457 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:02.894604 12457 heartbeater.cc:511] Master 127.0.249.126:33691 requested a full tablet report, sending...
I20260925 00:50:02.896405 12243 ts_manager.cc:194] Registered new tserver with Master: 0a1a977691904c9bb48e4eab5ed29727 (127.0.249.65:39031)
I20260925 00:50:02.897933 12243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:56819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:50:02.908581 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:56121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33691
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:50:02 dist-test-slave-05m0 krb5kdc[11767](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 00:50:03.071892 12463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:03.072301 12463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:03.072454 12463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:03.078838 12463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:03.078959 12463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:03.079082 12463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:50:03.087241 12463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.249.84:56121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:03.088908 12463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:03.090291 12463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:03.102952 12469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:03.103130 12472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:03.103735 12463 server_base.cc:1034] running on GCE node
W20260925 00:50:03.103751 12470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:03.104215 12463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:03.105026 12463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:03.106223 12463 hybrid_clock.cc:648] HybridClock initialized: now 1790297403106186 us; error 52 us; skew 500 ppm
Sep 25 00:50:03 dist-test-slave-05m0 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:03.109794 12463 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:50:03.110901 12463 webserver.cc:460] Webserver started at http://127.0.249.66:43137/ using document root <none> and password file <none>
I20260925 00:50:03.111599 12463 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:03.111690 12463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:03.111995 12463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:50:03 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:03.132891 12463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "f0cfa9a06cf04c1ea2d72d9f29152668"
format_stamp: "Formatted at 2026-09-25 00:50:03 on dist-test-slave-05m0"
server_key: "60f19bdd9fe9b0ae9b4c13a7089c1272"
server_key_iv: "377082058f6cc4105414fb55a5d2282c"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:03.133767 12463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "f0cfa9a06cf04c1ea2d72d9f29152668"
format_stamp: "Formatted at 2026-09-25 00:50:03 on dist-test-slave-05m0"
server_key: "60f19bdd9fe9b0ae9b4c13a7089c1272"
server_key_iv: "377082058f6cc4105414fb55a5d2282c"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:03.139333 12463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260925 00:50:03.161276 12479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:03.163182 12463 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 00:50:03.163337 12463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "f0cfa9a06cf04c1ea2d72d9f29152668"
format_stamp: "Formatted at 2026-09-25 00:50:03 on dist-test-slave-05m0"
server_key: "60f19bdd9fe9b0ae9b4c13a7089c1272"
server_key_iv: "377082058f6cc4105414fb55a5d2282c"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:03.163481 12463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:03.201177 12463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:03.207046 12463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:03.207219 12463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:03.207819 12463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:03.209024 12463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:03.209086 12463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:03.209159 12463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:03.209195 12463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:03.221467 12463 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:33407
I20260925 00:50:03.221657 12592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:33407 every 8 connection(s)
I20260925 00:50:03.222566 12463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:50:03.225373 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 12463
I20260925 00:50:03.225535 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:50:03.225898 997 external_mini_cluster.cc:1428] Setting key 4adbb1f7b5c39a84b166398d22b63858
Sep 25 00:50:03 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
I20260925 00:50:03.263947 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.224821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:36985 (local address 127.0.249.126:33691)
0925 00:50:03.224983 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.224988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.227055 (+ 2067us) server_negotiation.cc:407] Connection header received
0925 00:50:03.230139 (+ 3084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.230144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.230226 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.230295 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:03.232244 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.232800 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.234694 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.235131 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.235243 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.238588 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:03.238594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:03.238598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:03.238627 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:03.240612 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:03.250016 (+ 9404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:03.250021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:03.250023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:03.250143 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:03.250712 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:03.250716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:03.250718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:03.250960 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:03.251046 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.263558 (+ 12512us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.263789 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:50:03.265162 12595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.225365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:33691 (local address 127.0.249.66:36985)
0925 00:50:03.226901 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.226949 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:03.229819 (+ 2870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:03.230717 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:03.230790 (+ 73us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:03.231327 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:03.232071 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.232088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.232943 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.232950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.234531 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.234547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.235500 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.235503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.235660 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.236347 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:03.236374 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:03.238360 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:03.240742 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:03.240749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:03.240781 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:03.249830 (+ 9049us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:03.250271 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:03.250274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:03.250276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:03.250590 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:03.262586 (+ 11996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:03.262598 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:03.263133 (+ 535us) client_negotiation.cc:769] Sending connection context
0925 00:50:03.263434 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 00:50:03.263876 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1250,"thread_start_us":151,"threads_started":1}
I20260925 00:50:03.267515 12593 heartbeater.cc:347] Connected to a master server at 127.0.249.126:33691
I20260925 00:50:03.267797 12593 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:03.269352 12593 heartbeater.cc:511] Master 127.0.249.126:33691 requested a full tablet report, sending...
I20260925 00:50:03.270666 12243 ts_manager.cc:194] Registered new tserver with Master: f0cfa9a06cf04c1ea2d72d9f29152668 (127.0.249.66:33407)
I20260925 00:50:03.271266 12243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:36985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:50:03.272724 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:56121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33691
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:50:03 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
W20260925 00:50:03.431702 12598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:03.432024 12598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:03.432101 12598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:03.437309 12598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:03.437424 12598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:03.437534 12598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:50:03.444149 12598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.249.84:56121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:03.445767 12598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:03.447285 12598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:03.460919 12604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:03.461342 12598 server_base.cc:1034] running on GCE node
W20260925 00:50:03.461567 12606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:03.464162 12603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:03.464720 12598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:03.465582 12598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:03.466773 12598 hybrid_clock.cc:648] HybridClock initialized: now 1790297403466696 us; error 91 us; skew 500 ppm
Sep 25 00:50:03 dist-test-slave-05m0 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:03.473158 12598 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:50:03.474442 12598 webserver.cc:460] Webserver started at http://127.0.249.67:46481/ using document root <none> and password file <none>
I20260925 00:50:03.475214 12598 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:03.475371 12598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:03.475656 12598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:50:03 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:03.500881 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "554bc89072a34ef89e2fa51d8b0c6456"
format_stamp: "Formatted at 2026-09-25 00:50:03 on dist-test-slave-05m0"
server_key: "ea2f7dfd663203d2262718b22fe420f0"
server_key_iv: "d470b3b531e7818acd9c3c155aa2c712"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:03.501621 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "554bc89072a34ef89e2fa51d8b0c6456"
format_stamp: "Formatted at 2026-09-25 00:50:03 on dist-test-slave-05m0"
server_key: "ea2f7dfd663203d2262718b22fe420f0"
server_key_iv: "d470b3b531e7818acd9c3c155aa2c712"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:03.519945 12598 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.003s sys 0.001s
I20260925 00:50:03.545797 12613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:03.547555 12598 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 00:50:03.547693 12598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "554bc89072a34ef89e2fa51d8b0c6456"
format_stamp: "Formatted at 2026-09-25 00:50:03 on dist-test-slave-05m0"
server_key: "ea2f7dfd663203d2262718b22fe420f0"
server_key_iv: "d470b3b531e7818acd9c3c155aa2c712"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:03.547838 12598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:03.584290 12598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:03.597141 12598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:03.597354 12598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:03.598048 12598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:03.599046 12598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:03.599104 12598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:03.599161 12598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:03.599186 12598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:03.611550 12598 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:36559
I20260925 00:50:03.612988 12598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:50:03.618084 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 12598
I20260925 00:50:03.618207 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:50:03.618578 997 external_mini_cluster.cc:1428] Setting key c00557d74c1829f80c0d329805ce0ada
I20260925 00:50:03.614439 12726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:36559 every 8 connection(s)
Sep 25 00:50:03 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:03.635041 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.619549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:40787 (local address 127.0.249.126:33691)
0925 00:50:03.619800 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.619805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.620449 (+ 644us) server_negotiation.cc:407] Connection header received
0925 00:50:03.621772 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.621776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.621845 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.621923 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:03.623495 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.624265 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.625943 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.626268 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.626352 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.629282 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:03.629286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:03.629289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:03.629323 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:03.631033 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:03.633224 (+ 2191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:03.633228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:03.633230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:03.633341 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:03.633779 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:03.633782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:03.633784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:03.633995 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:03.634060 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.634765 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.634886 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260925 00:50:03.637328 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.616697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:33691 (local address 127.0.249.67:40787)
0925 00:50:03.620290 (+ 3593us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.620349 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:03.621519 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:03.622255 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:03.622265 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:03.622611 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:03.623331 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.623351 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.624429 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.624432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.625810 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.625821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.626543 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.626546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.626734 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.627476 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:03.627497 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:03.629083 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:03.631171 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:03.631177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:03.631195 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:03.633106 (+ 1911us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:03.633496 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:03.633500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:03.633502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:03.633679 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:03.634137 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:03.634144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:03.634486 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 00:50:03.635254 (+ 768us) client_negotiation.cc:240] Negotiation successful
0925 00:50:03.635652 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3347,"thread_start_us":128,"threads_started":1}
I20260925 00:50:03.638741 12727 heartbeater.cc:347] Connected to a master server at 127.0.249.126:33691
I20260925 00:50:03.639022 12727 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:03.639657 12727 heartbeater.cc:511] Master 127.0.249.126:33691 requested a full tablet report, sending...
I20260925 00:50:03.640797 12243 ts_manager.cc:194] Registered new tserver with Master: 554bc89072a34ef89e2fa51d8b0c6456 (127.0.249.67:36559)
I20260925 00:50:03.641607 12243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:40787
I20260925 00:50:03.644346 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:50:03.659129 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.646744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.0.249.126:33691)
0925 00:50:03.646857 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.646862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.649700 (+ 2838us) server_negotiation.cc:407] Connection header received
0925 00:50:03.649889 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.649892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.649937 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.649984 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:03.650956 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.651719 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.652953 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.653284 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.653361 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.655427 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:03.655431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:03.655434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:03.655460 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:03.657210 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:03.657780 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:03.657784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:03.657787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:03.657875 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:03.658197 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:03.658200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:03.658202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:03.658426 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:03.658552 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.658837 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.658958 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 00:50:03.663182 12243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:50:03.667591 12243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:50:03.690426 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.677629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:36559 (local address 127.0.0.1:60044)
0925 00:50:03.682432 (+ 4803us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.682481 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:03.682594 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:03.683512 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:03.683518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:03.683540 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:03.683661 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.683667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.685843 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.685849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.687927 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.687937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.689206 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.689210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.689327 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.689375 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 00:50:03.689488 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 00:50:03.690274 (+ 786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4624,"thread_start_us":89,"threads_started":1}
I20260925 00:50:03.691396 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.678428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.0.249.67:36559)
0925 00:50:03.682868 (+ 4440us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.682879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.682903 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:50:03.682976 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.682981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.683181 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.683342 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:03.684534 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.685721 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.688086 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.688917 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.689016 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.689152 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.691109 (+ 1957us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.691240 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4334,"thread_start_us":82,"threads_started":1}
I20260925 00:50:03.695259 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.680860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:33407 (local address 127.0.0.1:43246)
0925 00:50:03.685790 (+ 4930us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.685804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:03.685884 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:03.686703 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:03.686706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:03.686723 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:03.686807 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.686824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.688180 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.688183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.689259 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.689267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.694905 (+ 5638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.694908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.695007 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.695033 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:50:03.695078 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 00:50:03.695142 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4861,"thread_start_us":73,"threads_started":1}
I20260925 00:50:03.694900 12661 tablet_service.cc:1467] Processing CreateTablet for tablet 51ddb3b2319c42d9a997909f9348ca1a (DEFAULT_TABLE table=test-table [id=03900624611e4c4e8cf88aa67e318d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:03.696061 12661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ddb3b2319c42d9a997909f9348ca1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:03.696105 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.681203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.0.249.66:33407)
0925 00:50:03.686104 (+ 4901us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.686118 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.686140 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:50:03.686208 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.686213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.686404 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.686556 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:03.686953 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.688064 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.692506 (+ 4442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.694781 (+ 2275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.695691 (+ 910us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.695838 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.695887 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.695977 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4801,"thread_start_us":78,"threads_started":1}
I20260925 00:50:03.699128 12515 tablet_service.cc:1467] Processing CreateTablet for tablet 51ddb3b2319c42d9a997909f9348ca1a (DEFAULT_TABLE table=test-table [id=03900624611e4c4e8cf88aa67e318d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:03.700145 12515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ddb3b2319c42d9a997909f9348ca1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:03.705199 12745 tablet_bootstrap.cc:492] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: Bootstrap starting.
I20260925 00:50:03.708184 12745 tablet_bootstrap.cc:654] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:03.709096 12745 log.cc:824] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:03.709559 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.680086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:39031 (local address 127.0.0.1:43194)
0925 00:50:03.685525 (+ 5439us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.685538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:03.685617 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:03.694637 (+ 9020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:03.694640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:03.694657 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:03.694762 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.694768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.706785 (+ 12017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.706791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.707975 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.707987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.709079 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.709083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.709185 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.709219 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 00:50:03.709272 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 00:50:03.709352 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5379,"thread_start_us":91,"threads_started":1}
I20260925 00:50:03.710273 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.680469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.0.249.65:39031)
0925 00:50:03.694174 (+ 13705us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.694181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.694203 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:50:03.694300 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.694305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.694509 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.694653 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:03.705474 (+ 10821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.706629 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.708143 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.708929 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.709857 (+ 928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.710002 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.710053 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.710140 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13595,"thread_start_us":66,"threads_started":1}
I20260925 00:50:03.711265 12745 tablet_bootstrap.cc:492] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: No bootstrap required, opened a new log
I20260925 00:50:03.711458 12745 ts_tablet_manager.cc:1397] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:50:03.713127 12389 tablet_service.cc:1467] Processing CreateTablet for tablet 51ddb3b2319c42d9a997909f9348ca1a (DEFAULT_TABLE table=test-table [id=03900624611e4c4e8cf88aa67e318d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:03.714002 12746 tablet_bootstrap.cc:492] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: Bootstrap starting.
I20260925 00:50:03.714241 12389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ddb3b2319c42d9a997909f9348ca1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:03.715652 12745 raft_consensus.cc:348] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.715935 12745 raft_consensus.cc:374] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:03.715997 12745 raft_consensus.cc:729] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554bc89072a34ef89e2fa51d8b0c6456, State: Initialized, Role: FOLLOWER
I20260925 00:50:03.716688 12745 consensus_queue.cc:260] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.717813 12745 ts_tablet_manager.cc:1428] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 00:50:03.718173 12727 heartbeater.cc:503] Master 127.0.249.126:33691 was elected leader, sending a full tablet report...
I20260925 00:50:03.719269 12746 tablet_bootstrap.cc:654] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:03.723066 12746 log.cc:824] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:03.724574 12749 tablet_bootstrap.cc:492] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727: Bootstrap starting.
I20260925 00:50:03.727059 12749 tablet_bootstrap.cc:654] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:03.727942 12749 log.cc:824] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:03.730649 12749 tablet_bootstrap.cc:492] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727: No bootstrap required, opened a new log
I20260925 00:50:03.730867 12749 ts_tablet_manager.cc:1397] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 00:50:03.734403 12746 tablet_bootstrap.cc:492] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: No bootstrap required, opened a new log
I20260925 00:50:03.736172 12746 ts_tablet_manager.cc:1397] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20260925 00:50:03.736676 12749 raft_consensus.cc:348] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.736979 12749 raft_consensus.cc:374] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:03.737041 12749 raft_consensus.cc:729] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1a977691904c9bb48e4eab5ed29727, State: Initialized, Role: FOLLOWER
I20260925 00:50:03.737674 12749 consensus_queue.cc:260] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.738631 12748 raft_consensus.cc:482] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:50:03.738816 12748 raft_consensus.cc:504] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.740060 12748 leader_election.cc:288] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a1a977691904c9bb48e4eab5ed29727 (127.0.249.65:39031), f0cfa9a06cf04c1ea2d72d9f29152668 (127.0.249.66:33407)
I20260925 00:50:03.742377 12749 ts_tablet_manager.cc:1428] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260925 00:50:03.742753 12457 heartbeater.cc:503] Master 127.0.249.126:33691 was elected leader, sending a full tablet report...
I20260925 00:50:03.744004 12752 raft_consensus.cc:482] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:50:03.744131 12752 raft_consensus.cc:504] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.746274 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.741508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:43091 (local address 127.0.249.66:33407)
0925 00:50:03.741618 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.741623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.742012 (+ 389us) server_negotiation.cc:407] Connection header received
0925 00:50:03.742190 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.742193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.742236 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.742283 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:03.742673 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.743367 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.744914 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.745518 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.745608 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.745685 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.746060 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.746134 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:50:03.746917 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.740714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:33407 (local address 127.0.249.67:43091)
0925 00:50:03.741927 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.741941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:03.742071 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:03.742413 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:03.742416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:03.742435 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:03.742548 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.742553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.743506 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.743509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.744757 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.744768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.745802 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.745808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.745908 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.745936 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 00:50:03.746456 (+ 520us) client_negotiation.cc:240] Negotiation successful
0925 00:50:03.746533 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260925 00:50:03.747098 12752 leader_election.cc:288] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554bc89072a34ef89e2fa51d8b0c6456 (127.0.249.67:36559), f0cfa9a06cf04c1ea2d72d9f29152668 (127.0.249.66:33407)
I20260925 00:50:03.749017 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.740446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:39031 (local address 127.0.249.67:39439)
0925 00:50:03.741133 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.741148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:03.741240 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:03.741586 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:03.741590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:03.741608 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:03.741736 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.741743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.742986 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.742989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.747759 (+ 4770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.747769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.748589 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.748593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.748692 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.748715 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:50:03.748805 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 00:50:03.748888 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":3603}
I20260925 00:50:03.749471 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.741092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:39439 (local address 127.0.249.65:39031)
0925 00:50:03.741221 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.741226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.741245 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:50:03.741369 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.741372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.741448 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.741858 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:03.741957 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.742790 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.747913 (+ 5123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.748458 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.748567 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.748606 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.749271 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.749345 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:50:03.749905 12402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ddb3b2319c42d9a997909f9348ca1a" candidate_uuid: "554bc89072a34ef89e2fa51d8b0c6456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a1a977691904c9bb48e4eab5ed29727" is_pre_election: true
I20260925 00:50:03.747471 12547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ddb3b2319c42d9a997909f9348ca1a" candidate_uuid: "554bc89072a34ef89e2fa51d8b0c6456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" is_pre_election: true
I20260925 00:50:03.750092 12402 raft_consensus.cc:2454] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 554bc89072a34ef89e2fa51d8b0c6456 in term 0.
I20260925 00:50:03.750562 12614 leader_election.cc:302] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1a977691904c9bb48e4eab5ed29727, 554bc89072a34ef89e2fa51d8b0c6456; no voters:
I20260925 00:50:03.750874 12748 raft_consensus.cc:2781] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:50:03.750953 12748 raft_consensus.cc:482] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 00:50:03.750926 12615 leader_election.cc:341] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f0cfa9a06cf04c1ea2d72d9f29152668 (127.0.249.66:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260925 00:50:03.751003 12748 raft_consensus.cc:3037] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:03.752317 12748 raft_consensus.cc:504] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.752707 12748 leader_election.cc:288] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [CANDIDATE]: Term 1 election: Requested vote from peers 0a1a977691904c9bb48e4eab5ed29727 (127.0.249.65:39031), f0cfa9a06cf04c1ea2d72d9f29152668 (127.0.249.66:33407)
I20260925 00:50:03.753134 12402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ddb3b2319c42d9a997909f9348ca1a" candidate_uuid: "554bc89072a34ef89e2fa51d8b0c6456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a1a977691904c9bb48e4eab5ed29727"
I20260925 00:50:03.753176 12547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ddb3b2319c42d9a997909f9348ca1a" candidate_uuid: "554bc89072a34ef89e2fa51d8b0c6456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668"
I20260925 00:50:03.753247 12402 raft_consensus.cc:3037] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 0 FOLLOWER]: Advancing to term 1
W20260925 00:50:03.753700 12615 leader_election.cc:341] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f0cfa9a06cf04c1ea2d72d9f29152668 (127.0.249.66:33407): Illegal state: must be running to vote when last-logged opid is not known
I20260925 00:50:03.754608 12402 raft_consensus.cc:2454] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 554bc89072a34ef89e2fa51d8b0c6456 in term 1.
I20260925 00:50:03.754938 12614 leader_election.cc:302] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a1a977691904c9bb48e4eab5ed29727, 554bc89072a34ef89e2fa51d8b0c6456; no voters: f0cfa9a06cf04c1ea2d72d9f29152668
I20260925 00:50:03.755152 12748 raft_consensus.cc:2781] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:50:03.755380 12748 raft_consensus.cc:686] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 1 LEADER]: Becoming Leader. State: Replica: 554bc89072a34ef89e2fa51d8b0c6456, State: Running, Role: LEADER
I20260925 00:50:03.755723 12748 consensus_queue.cc:237] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.759001 12243 catalog_manager.cc:5495] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 reported cstate change: term changed from 0 to 1, leader changed from <none> to 554bc89072a34ef89e2fa51d8b0c6456 (127.0.249.67). New cstate: current_term: 1 leader_uuid: "554bc89072a34ef89e2fa51d8b0c6456" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260925 00:50:03.759297 12746 raft_consensus.cc:348] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.759663 12746 raft_consensus.cc:374] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:03.759743 12746 raft_consensus.cc:729] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0cfa9a06cf04c1ea2d72d9f29152668, State: Initialized, Role: FOLLOWER
I20260925 00:50:03.760267 12746 consensus_queue.cc:260] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bc89072a34ef89e2fa51d8b0c6456" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36559 } } peers { permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 } } peers { permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 } }
I20260925 00:50:03.761425 12746 ts_tablet_manager.cc:1428] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260925 00:50:03.761823 12593 heartbeater.cc:503] Master 127.0.249.126:33691 was elected leader, sending a full tablet report...
I20260925 00:50:03.764729 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.745902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:57423 (local address 127.0.249.67:36559)
0925 00:50:03.746013 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.746018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.748094 (+ 2076us) server_negotiation.cc:407] Connection header received
0925 00:50:03.748285 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.748290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.748331 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.748381 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:03.760872 (+ 12491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.761587 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.763468 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.763993 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.764085 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.764151 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.764547 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.764618 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 00:50:03.765069 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.745538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:36559 (local address 127.0.249.65:57423)
0925 00:50:03.748017 (+ 2479us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.748031 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:03.748167 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:03.760204 (+ 12037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:03.760207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:03.760231 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:03.760411 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.760420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.761720 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.761723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.763319 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.763330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.764299 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.764302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.764393 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.764416 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:50:03.764849 (+ 433us) client_negotiation.cc:240] Negotiation successful
0925 00:50:03.764923 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2406,"thread_start_us":104,"threads_started":1}
I20260925 00:50:03.765646 12662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ddb3b2319c42d9a997909f9348ca1a" candidate_uuid: "0a1a977691904c9bb48e4eab5ed29727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554bc89072a34ef89e2fa51d8b0c6456" is_pre_election: true
I20260925 00:50:03.770035 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.746381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:33407 (local address 127.0.249.65:57109)
0925 00:50:03.760644 (+ 14263us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.760658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:03.760748 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:03.762415 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:03.762417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:03.762435 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:03.766588 (+ 4153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.766595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.767478 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.767480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.768447 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:03.768457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.769442 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.769446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:03.769658 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.769687 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 00:50:03.769787 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 00:50:03.769853 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14188,"thread_start_us":86,"threads_started":1}
I20260925 00:50:03.770042 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.746673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:57109 (local address 127.0.249.66:33407)
0925 00:50:03.746775 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.746779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.761437 (+ 14658us) server_negotiation.cc:407] Connection header received
0925 00:50:03.761487 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.761489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.761547 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.761624 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:03.766724 (+ 5100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.767367 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.768581 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.769125 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.769220 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.769263 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.769835 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.769929 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 00:50:03.770680 12547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ddb3b2319c42d9a997909f9348ca1a" candidate_uuid: "0a1a977691904c9bb48e4eab5ed29727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" is_pre_election: true
I20260925 00:50:03.770927 12547 raft_consensus.cc:2454] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a1a977691904c9bb48e4eab5ed29727 in term 0.
I20260925 00:50:03.771242 12343 leader_election.cc:302] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a1a977691904c9bb48e4eab5ed29727, f0cfa9a06cf04c1ea2d72d9f29152668; no voters: 554bc89072a34ef89e2fa51d8b0c6456
I20260925 00:50:03.771786 12755 raft_consensus.cc:2741] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 00:50:03.774215 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.769453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.0.249.67:36559)
0925 00:50:03.769554 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:03.769557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:03.769656 (+ 99us) server_negotiation.cc:407] Connection header received
0925 00:50:03.769818 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:03.769821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:03.769881 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:03.769976 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:50:03.770349 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.771039 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.772258 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:03.772586 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:03.772678 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:03.773084 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:50:03.773583 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:50:03.773686 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:03.774053 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 00:50:03.774103 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 00:50:03 dist-test-slave-05m0 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:03.780946 12547 raft_consensus.cc:3037] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:03.782366 12547 raft_consensus.cc:1261] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668 [term 1 FOLLOWER]: Refusing update from remote peer 554bc89072a34ef89e2fa51d8b0c6456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:03.783279 12758 consensus_queue.cc:1035] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0cfa9a06cf04c1ea2d72d9f29152668" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:50:03.789677 12402 raft_consensus.cc:1261] T 51ddb3b2319c42d9a997909f9348ca1a P 0a1a977691904c9bb48e4eab5ed29727 [term 1 FOLLOWER]: Refusing update from remote peer 554bc89072a34ef89e2fa51d8b0c6456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:03.790225 12758 consensus_queue.cc:1035] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a1a977691904c9bb48e4eab5ed29727" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:50:03.798110 12762 mvcc.cc:205] Tried to move back new op lower bound from 7333058165878706176 to 7333058165786542080. Current Snapshot: MvccSnapshot[applied={T|T < 7333058165878706176 or (T in {7333058165878706176})}]
I20260925 00:50:03.798542 12763 mvcc.cc:205] Tried to move back new op lower bound from 7333058165878706176 to 7333058165786542080. Current Snapshot: MvccSnapshot[applied={T|T < 7333058165878706176}]
I20260925 00:50:03.806396 12242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:50:03.806559 12242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:50:03.807827 12765 mvcc.cc:205] Tried to move back new op lower bound from 7333058165878706176 to 7333058165786542080. Current Snapshot: MvccSnapshot[applied={T|T < 7333058165878706176}]
I20260925 00:50:03.809952 12242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd2aaaf1f14c4ace83401db439ff7f67: Sending DeleteTablet for 3 replicas of tablet 51ddb3b2319c42d9a997909f9348ca1a
I20260925 00:50:03.811045 12389 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ddb3b2319c42d9a997909f9348ca1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:03 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:43194
I20260925 00:50:03.811132 12661 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ddb3b2319c42d9a997909f9348ca1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:03 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:60044
I20260925 00:50:03.811470 12390 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ddb3b2319c42d9a997909f9348ca1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:03 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:43194
I20260925 00:50:03.811693 12515 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ddb3b2319c42d9a997909f9348ca1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:03 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:43246
I20260925 00:50:03.812855 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 12324
I20260925 00:50:03.817044 12772 tablet_replica.cc:330] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: stopping tablet replica
I20260925 00:50:03.817502 12772 raft_consensus.cc:2229] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:50:03.817777 12773 tablet_replica.cc:330] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: stopping tablet replica
I20260925 00:50:03.818177 12772 raft_consensus.cc:2258] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:50:03.818519 12773 raft_consensus.cc:2229] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:50:03.819069 12773 raft_consensus.cc:2258] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:50:03.820601 12773 ts_tablet_manager.cc:1905] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:50:03.820808 12772 ts_tablet_manager.cc:1905] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:50:03.823468 12773 ts_tablet_manager.cc:1918] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:50:03.823468 12772 ts_tablet_manager.cc:1918] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:50:03.823567 12772 log.cc:1196] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/51ddb3b2319c42d9a997909f9348ca1a
I20260925 00:50:03.823606 12773 log.cc:1196] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/51ddb3b2319c42d9a997909f9348ca1a
I20260925 00:50:03.824007 12773 ts_tablet_manager.cc:1939] T 51ddb3b2319c42d9a997909f9348ca1a P f0cfa9a06cf04c1ea2d72d9f29152668: Deleting consensus metadata
I20260925 00:50:03.825368 12228 catalog_manager.cc:4841] TS f0cfa9a06cf04c1ea2d72d9f29152668 (127.0.249.66:33407): tablet 51ddb3b2319c42d9a997909f9348ca1a (table test-table [id=03900624611e4c4e8cf88aa67e318d00]) successfully deleted
I20260925 00:50:03.833806 12772 ts_tablet_manager.cc:1939] T 51ddb3b2319c42d9a997909f9348ca1a P 554bc89072a34ef89e2fa51d8b0c6456: Deleting consensus metadata
W20260925 00:50:03.834097 12227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:39031 (error 108)
I20260925 00:50:03.835213 12228 catalog_manager.cc:4841] TS 554bc89072a34ef89e2fa51d8b0c6456 (127.0.249.67:36559): tablet 51ddb3b2319c42d9a997909f9348ca1a (table test-table [id=03900624611e4c4e8cf88aa67e318d00]) successfully deleted
I20260925 00:50:03.835753 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.835412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:39031 (local address 127.0.0.1:43200)
0925 00:50:03.835597 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.835668 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 00:50:03.835886 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 12463
W20260925 00:50:03.837184 12227 catalog_manager.cc:4584] TS 0a1a977691904c9bb48e4eab5ed29727 (127.0.249.65:39031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51ddb3b2319c42d9a997909f9348ca1a: Network error: Client connection negotiation failed: client connection to 127.0.249.65:39031: connect: Connection refused (error 111)
I20260925 00:50:03.838088 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:03.837801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:39031 (local address 127.0.0.1:43214)
0925 00:50:03.837963 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:03.838015 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 00:50:03.844556 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 12598
I20260925 00:50:03.851249 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 12211
I20260925 00:50:03.858433 997 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 00:50:04.213587 997 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 00:50:04.213713 997 mini_ranger.cc:66] Stopping Ranger...
I20260925 00:50:04.558526 997 mini_ranger.cc:68] Stopped Ranger
2026-09-25 00:50:04.558 UTC [11797] LOG: received smart shutdown request
2026-09-25 00:50:04.562 UTC [11797] LOG: background worker "logical replication launcher" (PID 11806) exited with exit code 1
2026-09-25 00:50:04.562 UTC [11801] LOG: shutting down
2026-09-25 00:50:04.735 UTC [11797] LOG: database system is shut down
2026-09-25T00:50:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46362 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:50:04 dist-test-slave-05m0 krb5kdc[12784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:50:04 dist-test-slave-05m0 krb5kdc[12784](info): set up 2 sockets
Sep 25 00:50:04 dist-test-slave-05m0 krb5kdc[12784](info): commencing operation
krb5kdc: starting...
W20260925 00:50:05.913516 997 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:50:05 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:50:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.249.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.249.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/rangeradmin_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/rangeradmin_127.0.249.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.249.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.249.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/rangerlookup_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/rangerlookup_127.0.249.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.249.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.249.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/HTTP_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/HTTP_127.0.249.84@KRBTEST.COM.keytab.
I20260925 00:50:06.076009 997 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyp5H6g/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/postgres -l logfile start
2026-09-25 00:50:08.561 UTC [12814] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 00:50:08.561 UTC [12814] LOG: listening on IPv4 address "127.0.249.84", port 50255
2026-09-25 00:50:08.565 UTC [12814] LOG: listening on Unix socket "/tmp/.s.PGSQL.50255"
2026-09-25 00:50:08.586 UTC [12817] LOG: database system was shut down at 2026-09-25 00:50:07 UTC
2026-09-25 00:50:08.590 UTC [12814] LOG: database system is ready to accept connections
I20260925 00:50:09.553320 997 mini_postgres.cc:96] Postgres bound to 50255
2026-09-25 00:50:09.559 UTC [12825] FATAL: database "slave" does not exist
127.0.249.84:50255 - accepting connections
I20260925 00:50:09.559947 997 mini_ranger.cc:161] Starting Ranger...
I20260925 00:50:09.574599 997 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 00:50:10.066177 997 mini_ranger.cc:87] Created ranger Postgres database
I20260925 00:50:10.066260 997 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-admin
2026-09-25 00:50:10,348 [I] DB FLAVOR :POSTGRES
2026-09-25 00:50:10,348 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 00:50:10,348 [I] Checking connection..
2026-09-25 00:50:10,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 00:50:10,611 [I] Checking connection passed.
2026-09-25 00:50:10,612 [I] --------- Verifying version history table ---------
2026-09-25 00:50:10,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:50:10,838 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 00:50:10,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:50:11,077 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 00:50:11,077 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 00:50:11,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 00:50:11.332 UTC [12889] WARNING: there is no transaction in progress
2026-09-25 00:50:11,339 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 00:50:11,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 00:50:11,593 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 00:50:11,593 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 00:50:11,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 00:50:11,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 00:50:12,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-05m0.c.gcp-upstream.internal','N') ;"
2026-09-25 00:50:12,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 00:50:12,560 [I] Table x_portal_user does not exist in database ranger
2026-09-25 00:50:12,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 00:50:12,785 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 00:50:12,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 00:50:13,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 00:50:13,274 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 00:50:13,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 00:50:14.269 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.285 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.305 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.320 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.336 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.353 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.410 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.416 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.430 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.440 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.450 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.463 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.472 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.478 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.484 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:14.490 UTC [12997] WARNING: there is no transaction in progress
2026-09-25 00:50:15,095 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 00:50:15,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 00:50:15,322 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 00:50:15,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 00:50:15,591 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 00:50:15,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 00:50:15,831 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 00:50:15,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 00:50:16,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 00:50:16,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-05m0.c.gcp-upstream.internal';"
2026-09-25 00:50:16,533 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 00:50:16,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 00:50:16,788 [I] DB_PATCHES have already been applied
I20260925 00:50:16.794230 997 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyp5H6g/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 00:50:16.794312 997 mini_ranger.cc:193] Using host: 127.0.249.84
I20260925 00:50:16.797619 997 mini_ranger.cc:227] Ranger admin URL: http://127.0.249.84:38705
Sep 25, 2026 12:50:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:50:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:50:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 12:50:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 12:50:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 12:50:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 12:50:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 00:50:17.924207 997 mini_ranger.cc:160] Time spent starting Ranger: real 8.364s user 0.000s sys 0.009s
Sep 25, 2026 12:50:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 12:50:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 12:50:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 12:50:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 12:50:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 00:50:37.742079 997 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.249.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.249.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/rangerkms_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.249.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/rangerkms_127.0.249.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.249.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.249.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.249.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/HTTP_127.0.249.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.249.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/HTTP_127.0.249.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 00:50:37 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297437, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 00:50:37.827689 997 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 00:50:37.837918 997 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 00:50:38.690349 997 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 00:50:38.690433 997 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-kms
I20260925 00:50:38.690456 997 mini_ranger_kms.cc:227] Using postgres at 127.0.249.84:50255
2026-09-25 00:50:38,854 [I] DB FLAVOR :POSTGRES
2026-09-25 00:50:38,855 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 00:50:38,855 [I] Checking connection
2026-09-25 00:50:38,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 00:50:39,099 [I] connection success
2026-09-25 00:50:39,100 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 00:50:39,100 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 00:50:39,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 00:50:39,326 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 00:50:39,326 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 00:50:39,326 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 00:50:39,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.249.84:50255/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 00:50:39,600 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 00:50:40.039016 997 mini_ranger_kms.cc:310] Created kms service
I20260925 00:50:40.227505 997 mini_ranger_kms.cc:326] Created kudu user
I20260925 00:50:40.268615 997 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 00:50:40 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297437, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:40.658085 997 mini_ranger_kms.cc:384] Added ranger policy
I20260925 00:50:40.658334 997 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyp5H6g/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyp5H6g/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 00:50:40.658370 997 mini_ranger_kms.cc:240] Using host: 127.0.249.84
I20260925 00:50:40.663394 997 mini_ranger_kms.cc:276] Ranger KMS PID: 13177
I20260925 00:50:40.663479 997 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.249.84:37509
Sep 25, 2026 12:50:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:50:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:50:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 12:50:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 12:50:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 12:50:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 00:50:42.879935 997 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.052s user 0.011s sys 0.138s
I20260925 00:50:42.880071 997 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 00:50:42.880151 997 mini_ranger_kms.cc:401] 127.0.249.84:37509/kms/v1/keys
Sep 25 00:50:43 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297443, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.249.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 00:50:43 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297443, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.249.84@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
Sep 25 00:50:43 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297443, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.249.84@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
Sep 25 00:50:46 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:50:46.183992 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:40937
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:40937
--ranger_config_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:37509/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:46.300647 13228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:46.300928 13228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:46.300966 13228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:46.305051 13228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:46.305107 13228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:46.305127 13228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:46.305143 13228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:46.305158 13228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:46.311005 13228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.249.84:37509/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:40937
--ranger_config_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:46.312076 13228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:46.313125 13228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:46.322402 13234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:46.322412 13233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:46.322606 13228 server_base.cc:1034] running on GCE node
W20260925 00:50:46.322412 13236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:46.323386 13228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:46.324445 13228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:46.325624 13228 hybrid_clock.cc:648] HybridClock initialized: now 1790297446325591 us; error 52 us; skew 500 ppm
Sep 25 00:50:46 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:46.328495 13228 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:50:46.329588 13228 webserver.cc:460] Webserver started at http://127.0.249.126:42097/ using document root <none> and password file <none>
I20260925 00:50:46.330120 13228 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:46.330173 13228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:46.330385 13228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:50:46 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:46.377223 13228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "7964dab02b76435e81ae927203016511"
format_stamp: "Formatted at 2026-09-25 00:50:46 on dist-test-slave-05m0"
server_key: "488ac6e9e22f9e74c7867c4f378c5110"
server_key_iv: "bf39658f75a69e8117c8494fa04666e2"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:46.378005 13228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "7964dab02b76435e81ae927203016511"
format_stamp: "Formatted at 2026-09-25 00:50:46 on dist-test-slave-05m0"
server_key: "488ac6e9e22f9e74c7867c4f378c5110"
server_key_iv: "bf39658f75a69e8117c8494fa04666e2"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:46.382946 13228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 00:50:46.399382 13243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:46.401129 13228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:50:46.401284 13228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "7964dab02b76435e81ae927203016511"
format_stamp: "Formatted at 2026-09-25 00:50:46 on dist-test-slave-05m0"
server_key: "488ac6e9e22f9e74c7867c4f378c5110"
server_key_iv: "bf39658f75a69e8117c8494fa04666e2"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:46.401463 13228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:46.428377 13228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:46.432372 13228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:46.432631 13228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:46.440703 13228 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:40937
I20260925 00:50:46.440721 13295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:40937 every 8 connection(s)
I20260925 00:50:46.441752 13228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:50:46.445269 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:46.451524 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 13228
I20260925 00:50:46.451659 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:50:46.451951 997 external_mini_cluster.cc:1428] Setting key 62a0ecc3c805b45eedac56651da67b3a
I20260925 00:50:46.452747 13296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Bootstrap starting.
I20260925 00:50:46.455655 13296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:46.456681 13296 log.cc:824] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:46.460186 13296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: No bootstrap required, opened a new log
Sep 25 00:50:46 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:46.464938 13296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } }
I20260925 00:50:46.465307 13296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:46.465477 13296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7964dab02b76435e81ae927203016511, State: Initialized, Role: FOLLOWER
I20260925 00:50:46.466032 13296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } }
I20260925 00:50:46.466182 13296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:50:46.466270 13296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:50:46.466368 13296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:46.466497 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:46.453476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.0.249.126:40937)
0925 00:50:46.453900 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:46.453918 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:46.453952 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:50:46.454565 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:46.454582 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:46.454912 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:46.455364 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:46.456144 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:46.457364 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:46.458658 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:46.459159 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:46.459337 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:46.461881 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:46.461887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:46.461900 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:46.461926 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:46.463574 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:46.464012 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:46.464026 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:46.464030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:46.464128 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:46.464428 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:46.464431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:46.464433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:46.464804 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:46.465027 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:46.465254 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 00:50:46.465583 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":137,"threads_started":1}
I20260925 00:50:46.467466 13296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } }
I20260925 00:50:46.467875 13296 leader_election.cc:302] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7964dab02b76435e81ae927203016511; no voters:
I20260925 00:50:46.468221 13296 leader_election.cc:288] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:50:46.468307 13301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:50:46.468567 13301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 1 LEADER]: Becoming Leader. State: Replica: 7964dab02b76435e81ae927203016511, State: Running, Role: LEADER
I20260925 00:50:46.468915 13301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } }
I20260925 00:50:46.469599 13296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:50:46.470194 13303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7964dab02b76435e81ae927203016511. Latest consensus state: current_term: 1 leader_uuid: "7964dab02b76435e81ae927203016511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } } }
I20260925 00:50:46.470403 13303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:46.470613 13302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7964dab02b76435e81ae927203016511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } } }
I20260925 00:50:46.470701 13302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:46.471159 13310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:50:46.474328 13310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:50:46.474723 13296 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 00:50:46.486742 13310 catalog_manager.cc:1347] Generated new cluster ID: f685ec9e4cae4f97bd736b7035327b45
I20260925 00:50:46.486816 13310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:50:46.501556 13310 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:50:46.502970 13310 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:50:46.513175 13310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Generated new TSK 0
I20260925 00:50:46.514097 13310 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:50:47.281507 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:37509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40937
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:47.430972 13341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:47.431313 13341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:47.431380 13341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:47.437337 13341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:47.437430 13341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:47.437517 13341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:50:47.445473 13341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.249.84:37509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:47.447000 13341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:47.448387 13341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:47.460176 13351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:47.460182 13348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:47.460698 13349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:47.461321 13341 server_base.cc:1034] running on GCE node
I20260925 00:50:47.461817 13341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:47.462585 13341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:47.463788 13341 hybrid_clock.cc:648] HybridClock initialized: now 1790297447463755 us; error 47 us; skew 500 ppm
Sep 25 00:50:47 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:47.467445 13341 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:50:47.468456 13341 webserver.cc:460] Webserver started at http://127.0.249.65:41989/ using document root <none> and password file <none>
I20260925 00:50:47.469135 13341 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:47.469220 13341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:47.469460 13341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:50:47 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:47.485786 13341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "038942e1595e47358520831c775e70b6"
format_stamp: "Formatted at 2026-09-25 00:50:47 on dist-test-slave-05m0"
server_key: "91b4a582f911a81e32ea4c66d871509e"
server_key_iv: "839b9d01c58f0475273a1b71b8874628"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:47.486572 13341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "038942e1595e47358520831c775e70b6"
format_stamp: "Formatted at 2026-09-25 00:50:47 on dist-test-slave-05m0"
server_key: "91b4a582f911a81e32ea4c66d871509e"
server_key_iv: "839b9d01c58f0475273a1b71b8874628"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:47.491712 13341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 00:50:47.519968 13358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:47.525957 13341 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.003s sys 0.001s
I20260925 00:50:47.526108 13341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "038942e1595e47358520831c775e70b6"
format_stamp: "Formatted at 2026-09-25 00:50:47 on dist-test-slave-05m0"
server_key: "91b4a582f911a81e32ea4c66d871509e"
server_key_iv: "839b9d01c58f0475273a1b71b8874628"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:47.526276 13341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:47.539800 13341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:47.543365 13341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:47.543593 13341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:47.544265 13341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:47.545116 13341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:47.545192 13341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:47.545295 13341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:47.545408 13341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:47.555892 13341 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:34611
I20260925 00:50:47.557104 13341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:50:47.558388 13473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:34611 every 8 connection(s)
I20260925 00:50:47.563373 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 13341
I20260925 00:50:47.563503 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:50:47.563820 997 external_mini_cluster.cc:1428] Setting key bb9e8fa8d33b823418c0664cf25b7ab4
Sep 25 00:50:47 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:50:47.580978 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:47.562965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:53849 (local address 127.0.249.126:40937)
0925 00:50:47.563704 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:47.563709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:47.567186 (+ 3477us) server_negotiation.cc:407] Connection header received
0925 00:50:47.568438 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:47.568441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:47.568499 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:47.568590 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:47.570261 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:47.571132 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:47.572492 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:47.572788 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:47.572912 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:47.576170 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:47.576177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:47.576183 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:47.576217 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:47.578342 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:47.579005 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:47.579009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:47.579010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:47.579103 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:47.579589 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:47.579594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:47.579596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:47.579835 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:47.579924 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:47.580653 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 00:50:47.580790 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":81,"threads_started":1}
I20260925 00:50:47.581941 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:47.561847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:40937 (local address 127.0.249.65:53849)
0925 00:50:47.564012 (+ 2165us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:47.564072 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:47.568052 (+ 3980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:47.568941 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:47.568951 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:47.569343 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:47.570048 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:47.570066 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:47.571425 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:47.571430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:47.572341 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:47.572349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:47.572969 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:47.572973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:47.573202 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:47.573959 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:47.573984 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:47.575949 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:47.578495 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:47.578504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:47.578525 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:47.578881 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:47.579251 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:47.579254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:47.579255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:47.579459 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:47.579978 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:47.579986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:47.580348 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 00:50:47.580597 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 00:50:47.580917 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1892,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
I20260925 00:50:47.583326 13474 heartbeater.cc:347] Connected to a master server at 127.0.249.126:40937
I20260925 00:50:47.583564 13474 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:47.584129 13474 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
I20260925 00:50:47.586063 13260 ts_manager.cc:194] Registered new tserver with Master: 038942e1595e47358520831c775e70b6 (127.0.249.65:34611)
I20260925 00:50:47.587743 13260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:53849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:50:47.600800 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:37509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40937
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:50:47 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 00:50:47.728327 13480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:47.728629 13480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:47.728708 13480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:47.733809 13480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:47.733894 13480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:47.733994 13480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:50:47.740108 13480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.249.84:37509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:47.741179 13480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:47.742306 13480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:47.752048 13486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:47.752281 13487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:47.752665 13489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:47.752904 13480 server_base.cc:1034] running on GCE node
I20260925 00:50:47.753319 13480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:47.754119 13480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:47.755334 13480 hybrid_clock.cc:648] HybridClock initialized: now 1790297447755304 us; error 43 us; skew 500 ppm
Sep 25 00:50:47 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:47.758179 13480 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:50:47.759138 13480 webserver.cc:460] Webserver started at http://127.0.249.66:40953/ using document root <none> and password file <none>
I20260925 00:50:47.759717 13480 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:47.759809 13480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:47.760061 13480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:50:47 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:47.779510 13480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5"
format_stamp: "Formatted at 2026-09-25 00:50:47 on dist-test-slave-05m0"
server_key: "a0701ebd2f784d4be6a964d7b2e4cd86"
server_key_iv: "ad453d753a62a192d441f6822192f215"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:47.780349 13480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5"
format_stamp: "Formatted at 2026-09-25 00:50:47 on dist-test-slave-05m0"
server_key: "a0701ebd2f784d4be6a964d7b2e4cd86"
server_key_iv: "ad453d753a62a192d441f6822192f215"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:47.785530 13480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 00:50:47.800841 13496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:47.802498 13480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 00:50:47.802621 13480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5"
format_stamp: "Formatted at 2026-09-25 00:50:47 on dist-test-slave-05m0"
server_key: "a0701ebd2f784d4be6a964d7b2e4cd86"
server_key_iv: "ad453d753a62a192d441f6822192f215"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:47.802753 13480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:47.816222 13480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:47.819845 13480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:47.820082 13480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:47.820752 13480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:47.821810 13480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:47.821918 13480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:47.822016 13480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:47.822079 13480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:47.832135 13480 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:42405
I20260925 00:50:47.833230 13480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:50:47.835304 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 13480
I20260925 00:50:47.835412 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:50:47.835709 997 external_mini_cluster.cc:1428] Setting key 8a5a349705526761cc834efd98cee7ac
I20260925 00:50:47.846372 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:42405 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
Sep 25 00:50:47 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:47.864573 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:47.849587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:60207 (local address 127.0.249.126:40937)
0925 00:50:47.849714 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:47.849719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:47.851264 (+ 1545us) server_negotiation.cc:407] Connection header received
0925 00:50:47.852453 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:47.852457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:47.852508 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:47.852598 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:47.854131 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:47.854660 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:47.856406 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:47.856759 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:47.856838 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:47.859864 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:47.859868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:47.859871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:47.859900 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:47.861704 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:47.862678 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:47.862682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:47.862684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:47.862770 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:47.863198 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:47.863201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:47.863203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:47.863432 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:47.863528 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:47.864209 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 00:50:47.864405 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:50:47.865885 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:47.849138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:40937 (local address 127.0.249.66:60207)
0925 00:50:47.851084 (+ 1946us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:47.851142 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:47.852212 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:47.852911 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:47.852920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:47.853251 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:47.853938 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:47.853958 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:47.854835 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:47.854839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:47.856244 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:47.856255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:47.857037 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:47.857041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:47.857235 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:47.857969 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:47.857992 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:47.859665 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:47.861828 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:47.861836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:47.861852 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:47.862198 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:47.862887 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:47.862890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:47.862893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:47.863081 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:47.863561 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:47.863568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:47.863910 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 00:50:47.864131 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 00:50:47.864490 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1713,"thread_start_us":520,"threads_started":1}
I20260925 00:50:47.867573 13610 heartbeater.cc:347] Connected to a master server at 127.0.249.126:40937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:50:47.867812 13610 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:47.868389 13610 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
I20260925 00:50:47.868763 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:37509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40937
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:50:47.869836 13260 ts_manager.cc:194] Registered new tserver with Master: 75d71c3c8aea4a0ea52111c1a33fd8f5 (127.0.249.66:42405)
I20260925 00:50:47.870582 13260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:60207
W20260925 00:50:48.008288 13615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:48.008699 13615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:48.008795 13615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:48.013592 13615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:48.013667 13615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:48.013767 13615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:50:48.019170 13615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.249.84:37509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:48.020339 13615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:48.021507 13615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:48.032436 13623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:48.033356 13621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:48.034373 13615 server_base.cc:1034] running on GCE node
W20260925 00:50:48.042388 13620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:48.043002 13615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:48.043721 13615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:48.044930 13615 hybrid_clock.cc:648] HybridClock initialized: now 1790297448044890 us; error 49 us; skew 500 ppm
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:48.048691 13615 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:50:48.049793 13615 webserver.cc:460] Webserver started at http://127.0.249.67:46611/ using document root <none> and password file <none>
I20260925 00:50:48.050508 13615 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:48.050604 13615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:48.050856 13615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:48.070938 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "aa949257c4824fe5b8cdc116fcd9dd1a"
format_stamp: "Formatted at 2026-09-25 00:50:48 on dist-test-slave-05m0"
server_key: "8bcb7f73d19b73abb7a59ca1ee9f9173"
server_key_iv: "e454c17ade39d00e85001539b53d2839"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:48.071794 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "aa949257c4824fe5b8cdc116fcd9dd1a"
format_stamp: "Formatted at 2026-09-25 00:50:48 on dist-test-slave-05m0"
server_key: "8bcb7f73d19b73abb7a59ca1ee9f9173"
server_key_iv: "e454c17ade39d00e85001539b53d2839"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:48.076814 13615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 00:50:48.097164 13630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.098999 13615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:50:48.099208 13615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "aa949257c4824fe5b8cdc116fcd9dd1a"
format_stamp: "Formatted at 2026-09-25 00:50:48 on dist-test-slave-05m0"
server_key: "8bcb7f73d19b73abb7a59ca1ee9f9173"
server_key_iv: "e454c17ade39d00e85001539b53d2839"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:48.099452 13615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:48.128518 13615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:48.133667 13615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:48.133921 13615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:48.134649 13615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:48.135718 13615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:48.135804 13615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.135897 13615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:48.135950 13615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.145882 13615 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:42535
I20260925 00:50:48.147012 13615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:50:48.147122 13743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:42535 every 8 connection(s)
I20260925 00:50:48.147394 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 13615
I20260925 00:50:48.147503 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:50:48.147827 997 external_mini_cluster.cc:1428] Setting key a1e15559fbb159819d8fb68bc4b5bb59
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:48.170917 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:48.156142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:53751 (local address 127.0.249.126:40937)
0925 00:50:48.156262 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:48.156266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:48.158102 (+ 1836us) server_negotiation.cc:407] Connection header received
0925 00:50:48.159340 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:48.159344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:48.159398 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:48.159452 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:48.161050 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.161814 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.163458 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.163792 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.163878 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:48.166683 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:48.166687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:48.166689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:48.166716 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:48.168518 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:48.169168 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:48.169172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:48.169174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:48.169257 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:48.169679 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:48.169682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:48.169684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:48.169904 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:48.169971 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:48.170619 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 00:50:48.170745 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:50:48.172416 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:48.156716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:40937 (local address 127.0.249.67:53751)
0925 00:50:48.157952 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:48.158007 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:48.159101 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:48.159860 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:48.159868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:48.160232 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:48.160901 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:48.160921 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.161967 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.161970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:48.163317 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:48.163328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.164067 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.164071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:48.164255 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:48.164957 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:48.164977 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:48.166500 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:48.168655 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:48.168661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:48.168677 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:48.169037 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:48.169367 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:48.169370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:48.169372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:48.169568 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:48.170051 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:48.170057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:48.170355 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 00:50:48.171085 (+ 730us) client_negotiation.cc:240] Negotiation successful
0925 00:50:48.171414 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":173,"threads_started":1}
I20260925 00:50:48.173656 13744 heartbeater.cc:347] Connected to a master server at 127.0.249.126:40937
I20260925 00:50:48.173902 13744 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:48.174459 13744 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
I20260925 00:50:48.175601 13260 ts_manager.cc:194] Registered new tserver with Master: aa949257c4824fe5b8cdc116fcd9dd1a (127.0.249.67:42535)
I20260925 00:50:48.176316 13260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:53751
I20260925 00:50:48.185213 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:50:48.186167 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 13341
I20260925 00:50:48.193383 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 13480
I20260925 00:50:48.198423 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 13615
I20260925 00:50:48.205125 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 13228
I20260925 00:50:48.210811 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:40937
--webserver_interface=127.0.249.126
--webserver_port=42097
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:40937
--ranger_config_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:37509/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.249.84:37509/kms with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:48.323872 13747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:48.324172 13747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:48.324259 13747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:48.328625 13747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:48.328723 13747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:48.328763 13747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:48.328821 13747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:48.328869 13747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:48.334444 13747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.249.84:37509/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:40937
--ranger_config_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:48.335556 13747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:48.336617 13747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:48.345134 13753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:48.345142 13752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:48.345142 13755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:48.345908 13747 server_base.cc:1034] running on GCE node
I20260925 00:50:48.346525 13747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:48.347641 13747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:48.348932 13747 hybrid_clock.cc:648] HybridClock initialized: now 1790297448348903 us; error 50 us; skew 500 ppm
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:48.355850 13747 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:50:48.356927 13747 webserver.cc:460] Webserver started at http://127.0.249.126:42097/ using document root <none> and password file <none>
I20260925 00:50:48.357518 13747 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:48.357587 13747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:48.403322 13747 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 00:50:48.405748 13763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.407091 13747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 00:50:48.407228 13747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "7964dab02b76435e81ae927203016511"
format_stamp: "Formatted at 2026-09-25 00:50:46 on dist-test-slave-05m0"
server_key: "488ac6e9e22f9e74c7867c4f378c5110"
server_key_iv: "bf39658f75a69e8117c8494fa04666e2"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:48.407791 13747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:48.423022 13747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:48.426249 13747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:48.426460 13747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:48.434583 13747 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:40937
I20260925 00:50:48.434587 13815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:40937 every 8 connection(s)
I20260925 00:50:48.435729 13747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:50:48.437987 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 13747
I20260925 00:50:48.438441 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:37509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:34611
--local_ip_for_outbound_sockets=127.0.249.65
--tserver_master_addrs=127.0.249.126:40937
--webserver_port=41989
--webserver_interface=127.0.249.65
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.249.84:37509/kms with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:50:48.441181 13816 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:50:48.443805 13816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Bootstrap starting.
I20260925 00:50:48.466822 13816 log.cc:824] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:48.475044 13816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:50:48.475492 13816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Bootstrap complete.
I20260925 00:50:48.480908 13816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } }
I20260925 00:50:48.481426 13816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7964dab02b76435e81ae927203016511, State: Initialized, Role: FOLLOWER
I20260925 00:50:48.481969 13816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } }
I20260925 00:50:48.482112 13816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:50:48.482213 13816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:50:48.482318 13816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:50:48.484920 13816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } }
I20260925 00:50:48.485282 13816 leader_election.cc:302] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7964dab02b76435e81ae927203016511; no voters:
I20260925 00:50:48.485672 13816 leader_election.cc:288] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 00:50:48.485874 13820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:50:48.486158 13820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [term 2 LEADER]: Becoming Leader. State: Replica: 7964dab02b76435e81ae927203016511, State: Running, Role: LEADER
I20260925 00:50:48.486548 13820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } }
I20260925 00:50:48.487030 13816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:50:48.488085 13821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7964dab02b76435e81ae927203016511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } } }
I20260925 00:50:48.488059 13822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7964dab02b76435e81ae927203016511. Latest consensus state: current_term: 2 leader_uuid: "7964dab02b76435e81ae927203016511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7964dab02b76435e81ae927203016511" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40937 } } }
I20260925 00:50:48.488234 13822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:48.488234 13821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:48.488672 13829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:50:48.491983 13829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:50:48.492524 13816 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 00:50:48.492712 13829 catalog_manager.cc:1259] Loaded cluster ID: f685ec9e4cae4f97bd736b7035327b45
I20260925 00:50:48.492748 13829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:50:48.495234 13829 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:50:48.495831 13829 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Loaded TSK: 0
I20260925 00:50:48.496811 13829 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:50:48.596932 13818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:48.597265 13818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:48.597332 13818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:48.603359 13818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:48.603484 13818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:48.603593 13818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:50:48.611090 13818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.249.84:37509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:34611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=41989
--tserver_master_addrs=127.0.249.126:40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:48.612438 13818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:48.613739 13818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:48.625682 13856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:48.626194 13853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:48.626533 13854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:48.626636 13818 server_base.cc:1034] running on GCE node
I20260925 00:50:48.627211 13818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:48.628118 13818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:48.629354 13818 hybrid_clock.cc:648] HybridClock initialized: now 1790297448629333 us; error 35 us; skew 500 ppm
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:48.633087 13818 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:50:48.634313 13818 webserver.cc:460] Webserver started at http://127.0.249.65:41989/ using document root <none> and password file <none>
I20260925 00:50:48.635044 13818 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:48.635186 13818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:48.662010 13818 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260925 00:50:48.665494 13864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.667183 13818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 00:50:48.667330 13818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "038942e1595e47358520831c775e70b6"
format_stamp: "Formatted at 2026-09-25 00:50:47 on dist-test-slave-05m0"
server_key: "91b4a582f911a81e32ea4c66d871509e"
server_key_iv: "839b9d01c58f0475273a1b71b8874628"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:48.668089 13818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:48.695940 13818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:48.699831 13818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:48.700086 13818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:48.700785 13818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:48.701797 13818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:48.701886 13818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.701958 13818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:48.701995 13818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.711320 13818 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:34611
I20260925 00:50:48.712493 13818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:50:48.717625 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 13818
I20260925 00:50:48.718160 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:37509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:42405
--local_ip_for_outbound_sockets=127.0.249.66
--tserver_master_addrs=127.0.249.126:40937
--webserver_port=40953
--webserver_interface=127.0.249.66
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.249.84:37509/kms with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:50:48.717936 13977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:34611 every 8 connection(s)
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:48.740518 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:48.718905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:40937 (local address 127.0.249.65:37783)
0925 00:50:48.719711 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:48.719764 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:48.721345 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:48.724936 (+ 3591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:48.724945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:48.725278 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:48.725959 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:48.725979 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.727906 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.727911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:48.729481 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:48.729494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.730665 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.730670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:48.730900 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:48.731658 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:48.731681 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:48.733623 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:48.736485 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:48.736491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:48.736507 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:48.736905 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:48.737365 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:48.737368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:48.737370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:48.737598 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:48.738470 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:48.738477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:48.738787 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 00:50:48.739010 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 00:50:48.739359 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":154,"threads_started":1}
I20260925 00:50:48.743412 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:48.719094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:37783 (local address 127.0.249.126:40937)
0925 00:50:48.720851 (+ 1757us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:48.720882 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:48.720933 (+ 51us) server_negotiation.cc:407] Connection header received
0925 00:50:48.721969 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:48.721984 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:48.722388 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:48.722749 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:48.726155 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.727741 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.729625 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.730472 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.730670 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:48.733884 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:48.733891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:48.733910 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:48.733946 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:48.736337 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:48.737063 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:48.737070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:48.737076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:48.737236 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:48.737729 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:48.737732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:48.737735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:48.738257 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:48.738407 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:48.742005 (+ 3598us) server_negotiation.cc:299] Negotiation successful
0925 00:50:48.742336 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1571,"thread_start_us":105,"threads_started":1}
I20260925 00:50:48.746763 13978 heartbeater.cc:347] Connected to a master server at 127.0.249.126:40937
I20260925 00:50:48.747030 13978 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:48.747622 13978 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
W20260925 00:50:48.750097 13978 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:40937 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 00:50:48.750226 13978 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:48.750403 13978 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
I20260925 00:50:48.751457 13978 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:48.751629 13978 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
W20260925 00:50:48.752691 13978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:50:48.858139 13982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:48.858417 13982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:48.858515 13982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:48.862792 13982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:48.862870 13982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:48.862986 13982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:50:48.868355 13982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.249.84:37509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:42405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=40953
--tserver_master_addrs=127.0.249.126:40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:48.869365 13982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:48.870429 13982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:48.880263 13987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:48.880265 13988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:48.880292 13990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:48.880839 13982 server_base.cc:1034] running on GCE node
I20260925 00:50:48.881304 13982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:48.882109 13982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:48.883298 13982 hybrid_clock.cc:648] HybridClock initialized: now 1790297448883273 us; error 46 us; skew 500 ppm
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:48.886085 13982 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:50:48.886987 13982 webserver.cc:460] Webserver started at http://127.0.249.66:40953/ using document root <none> and password file <none>
I20260925 00:50:48.887604 13982 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:48.887758 13982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:48.903813 13982 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:50:48.906381 13998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.907665 13982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:50:48.907791 13982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5"
format_stamp: "Formatted at 2026-09-25 00:50:47 on dist-test-slave-05m0"
server_key: "a0701ebd2f784d4be6a964d7b2e4cd86"
server_key_iv: "ad453d753a62a192d441f6822192f215"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:48.908378 13982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:48.929065 13982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:48.932578 13982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:48.932782 13982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:48.933398 13982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:48.934419 13982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:48.934494 13982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.934582 13982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:48.934620 13982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:48.945345 13982 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:42405
I20260925 00:50:48.945642 14111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:42405 every 8 connection(s)
I20260925 00:50:48.946429 13982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:50:48.949002 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 13982
I20260925 00:50:48.949524 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.249.84:37509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:42535
--local_ip_for_outbound_sockets=127.0.249.67
--tserver_master_addrs=127.0.249.126:40937
--webserver_port=46611
--webserver_interface=127.0.249.67
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.249.84:37509/kms with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:50:48 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:48.962306 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:48.948387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:49187 (local address 127.0.249.126:40937)
0925 00:50:48.948858 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:48.948865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:48.949634 (+ 769us) server_negotiation.cc:407] Connection header received
0925 00:50:48.950959 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:48.950963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:48.951022 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:48.951081 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:48.952756 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.953607 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.954727 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.954969 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.955084 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:48.957555 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:48.957562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:48.957569 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:48.957607 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:48.959839 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:48.960483 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:48.960487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:48.960489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:48.960577 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:48.960975 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:48.960979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:48.960981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:48.961195 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:48.961262 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:48.961970 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 00:50:48.962110 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260925 00:50:48.963920 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:48.948825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:40937 (local address 127.0.249.66:49187)
0925 00:50:48.949463 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:48.949524 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:48.950697 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:48.951515 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:48.951525 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:48.951893 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:48.952571 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:48.952591 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.953731 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.953734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:48.954594 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:48.954603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:48.955122 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:48.955126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:48.955294 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:48.955886 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:48.955902 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:48.957305 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:48.959977 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:48.959984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:48.959999 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:48.960336 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:48.960690 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:48.960693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:48.960695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:48.960864 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:48.961357 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:48.961365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:48.961691 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 00:50:48.962575 (+ 884us) client_negotiation.cc:240] Negotiation successful
0925 00:50:48.962903 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":140,"threads_started":1}
I20260925 00:50:48.965499 14112 heartbeater.cc:347] Connected to a master server at 127.0.249.126:40937
I20260925 00:50:48.965785 14112 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:48.966468 14112 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
W20260925 00:50:48.968066 14112 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:40937 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 00:50:48.968230 14112 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:48.968437 14112 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
I20260925 00:50:48.969270 14112 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:48.969513 14112 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
W20260925 00:50:48.970250 14112 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:50:49.116109 14115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:49.116587 14115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:49.116674 14115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:49.122951 14115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:49.123037 14115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:49.123132 14115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:50:49.131212 14115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.249.84:37509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:42535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=46611
--tserver_master_addrs=127.0.249.126:40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:49.132644 14115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:49.134032 14115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:49.145751 14122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:49.145750 14124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:49.146227 14115 server_base.cc:1034] running on GCE node
W20260925 00:50:49.145865 14121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:49.146795 14115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:49.147511 14115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:49.150835 14115 hybrid_clock.cc:648] HybridClock initialized: now 1790297449150770 us; error 51 us; skew 500 ppm
Sep 25 00:50:49 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:49.154402 14115 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:50:49.155539 14115 webserver.cc:460] Webserver started at http://127.0.249.67:46611/ using document root <none> and password file <none>
I20260925 00:50:49.156100 14115 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:49.156203 14115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 00:50:49 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:49.177855 14115 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:50:49.181053 14132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:49.182672 14115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:50:49.182792 14115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "aa949257c4824fe5b8cdc116fcd9dd1a"
format_stamp: "Formatted at 2026-09-25 00:50:48 on dist-test-slave-05m0"
server_key: "8bcb7f73d19b73abb7a59ca1ee9f9173"
server_key_iv: "e454c17ade39d00e85001539b53d2839"
server_key_version: "kuduclusterkey@0"
I20260925 00:50:49.183499 14115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:49.208365 14115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:49.212298 14115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:49.212551 14115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:49.213258 14115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:49.214190 14115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:49.214262 14115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:49.214349 14115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:49.214395 14115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:49.223273 14115 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:42535
I20260925 00:50:49.223553 14245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:42535 every 8 connection(s)
I20260925 00:50:49.224372 14115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:50:49.234056 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 14115
Sep 25 00:50:49 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:49.240653 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:49.226471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:48643 (local address 127.0.249.126:40937)
0925 00:50:49.226597 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:49.226602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:49.228219 (+ 1617us) server_negotiation.cc:407] Connection header received
0925 00:50:49.229274 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:49.229278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:49.229335 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:49.229495 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:49.230875 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:49.231732 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:49.233473 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:49.233721 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:49.233847 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:49.237010 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:49.237014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:49.237016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:49.237057 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:49.238683 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:49.239197 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:49.239200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:49.239202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:49.239307 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:49.239634 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:49.239638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:49.239640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:49.239806 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:49.239906 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:49.240369 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 00:50:49.240489 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:50:49.241761 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:49.226973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:40937 (local address 127.0.249.67:48643)
0925 00:50:49.228063 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:49.228108 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:49.229049 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:49.229681 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:49.229688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:49.229973 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:49.230655 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:49.230686 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:49.231873 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:49.231876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:49.233300 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:49.233313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:49.234025 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:49.234031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:49.234245 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:49.235415 (+ 1170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:49.235431 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:49.236816 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:49.238793 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:49.238799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:49.238813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:49.239104 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:49.239399 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:49.239404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:49.239406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:49.239544 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:49.239905 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:49.239911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:49.240148 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 00:50:49.240330 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 00:50:49.240630 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":157,"threads_started":1}
I20260925 00:50:49.243227 14246 heartbeater.cc:347] Connected to a master server at 127.0.249.126:40937
I20260925 00:50:49.243520 14246 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:49.244082 14246 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
W20260925 00:50:49.246430 14246 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:40937 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 00:50:49.246564 14246 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:49.246747 14246 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
I20260925 00:50:49.247617 14246 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:49.247809 14246 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
W20260925 00:50:49.248611 14246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 00:50:49.249377 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:49.234732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.0.249.126:40937)
0925 00:50:49.235414 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:49.235425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:49.235439 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:50:49.235475 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:49.235477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:49.235523 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:49.235611 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:49.236641 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:49.237829 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:49.241481 (+ 3652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:49.241798 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:49.241914 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:49.244941 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:49.244946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:49.244948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:49.244971 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:49.246758 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:49.247397 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:49.247400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:49.247402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:49.247485 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:49.248445 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:49.248448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:49.248450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:49.248663 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:49.248836 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:49.249086 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 00:50:49.249201 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":102,"threads_started":1}
Sep 25 00:50:49 dist-test-slave-05m0 krb5kdc[12784](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790297449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:49.752969 13978 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:49.753307 13978 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
I20260925 00:50:49.754731 13780 ts_manager.cc:194] Registered new tserver with Master: 038942e1595e47358520831c775e70b6 (127.0.249.65:34611)
I20260925 00:50:49.756503 13780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:37783
I20260925 00:50:49.970527 14112 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:49.970826 14112 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
I20260925 00:50:49.971740 13780 ts_manager.cc:194] Registered new tserver with Master: 75d71c3c8aea4a0ea52111c1a33fd8f5 (127.0.249.66:42405)
I20260925 00:50:49.972488 13780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:49187
Sep 25 00:50:49 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
Sep 25 00:50:50 dist-test-slave-05m0 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for HTTP/127.0.249.84@KRBTEST.COM
I20260925 00:50:50.248854 14246 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:50.249188 14246 heartbeater.cc:511] Master 127.0.249.126:40937 requested a full tablet report, sending...
I20260925 00:50:50.250198 13780 ts_manager.cc:194] Registered new tserver with Master: aa949257c4824fe5b8cdc116fcd9dd1a (127.0.249.67:42535)
I20260925 00:50:50.250979 13780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:48643
I20260925 00:50:50.254912 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:50:50.271672 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.257673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.0.249.126:40937)
0925 00:50:50.258779 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:50.258784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:50.259831 (+ 1047us) server_negotiation.cc:407] Connection header received
0925 00:50:50.260223 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:50.260227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:50.260513 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:50.260779 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:50.261981 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.263705 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.265567 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.265898 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.265989 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.267709 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:50.267713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:50.267715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:50.267742 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:50.269604 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:50.270054 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:50.270057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:50.270060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:50.270138 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:50.270416 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:50.270420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:50.270422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:50.270619 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:50.270733 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:50.271374 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 00:50:50.271495 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":88,"threads_started":1}
I20260925 00:50:50.276333 13780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:50:50.280678 13780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:50:50.302623 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.295807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:42405 (local address 127.0.0.1:43616)
0925 00:50:50.297197 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:50.297210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:50.297310 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:50.297974 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:50.297977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:50.297994 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:50.298065 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.298071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.299385 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.299389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.300748 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.300758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.302071 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.302075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.302184 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.302234 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 00:50:50.302341 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 00:50:50.302468 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":95,"threads_started":1}
I20260925 00:50:50.303270 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.296122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.0.249.66:42405)
0925 00:50:50.297072 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:50.297078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:50.297417 (+ 339us) server_negotiation.cc:407] Connection header received
0925 00:50:50.297510 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:50.297517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:50.297714 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:50.297844 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:50.298193 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.299260 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.300880 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.301760 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.301855 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.302004 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:50.303022 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0925 00:50:50.303136 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":113,"threads_started":1}
I20260925 00:50:50.304080 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.295383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:34611 (local address 127.0.0.1:51560)
0925 00:50:50.296736 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:50.296746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:50.296849 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:50.297555 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:50.297561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:50.297582 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:50.297707 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.297714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.300231 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.300234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.301323 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.301333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.303683 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.303688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.303785 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.303815 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 00:50:50.303881 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 00:50:50.303949 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260925 00:50:50.304586 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.295559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.0.249.65:34611)
0925 00:50:50.296039 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:50.296047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:50.296841 (+ 794us) server_negotiation.cc:407] Connection header received
0925 00:50:50.296961 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:50.296966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:50.297198 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:50.297349 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:50.297846 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.300083 (+ 2237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.301497 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.303365 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.303470 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.303615 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:50.304332 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 00:50:50.304447 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":112,"threads_started":1}
I20260925 00:50:50.306957 14034 tablet_service.cc:1467] Processing CreateTablet for tablet ae76d4417d8241cdb620e5ea12a9a721 (DEFAULT_TABLE table=test-table [id=8e7157bd71bd40a3b4241a95cce5b966]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:50.308081 14034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae76d4417d8241cdb620e5ea12a9a721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:50.310968 13912 tablet_service.cc:1467] Processing CreateTablet for tablet ae76d4417d8241cdb620e5ea12a9a721 (DEFAULT_TABLE table=test-table [id=8e7157bd71bd40a3b4241a95cce5b966]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:50.311998 13912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae76d4417d8241cdb620e5ea12a9a721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:50.317750 14281 tablet_bootstrap.cc:492] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Bootstrap starting.
I20260925 00:50:50.321444 14281 tablet_bootstrap.cc:654] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:50.322407 14281 log.cc:824] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:50.324185 14276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.295053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42535 (local address 127.0.0.1:47424)
0925 00:50:50.296294 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:50.296342 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:50.296475 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:50.319248 (+ 22773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:50.319251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:50.319273 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:50.319403 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.319412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.321088 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.321092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.322219 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.322232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.323774 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.323778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.323878 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.323907 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 00:50:50.323957 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 00:50:50.324031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":107,"threads_started":1}
I20260925 00:50:50.324649 14281 tablet_bootstrap.cc:492] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: No bootstrap required, opened a new log
I20260925 00:50:50.324851 14281 ts_tablet_manager.cc:1397] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 00:50:50.325731 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.296200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.249.67:42535)
0925 00:50:50.318765 (+ 22565us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:50.318773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:50.318800 (+ 27us) server_negotiation.cc:407] Connection header received
0925 00:50:50.318880 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:50.318885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:50.319080 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:50.319592 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:50.319768 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.320892 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.322756 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.323615 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.325237 (+ 1622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.325411 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:50.325468 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 00:50:50.325564 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22425,"thread_start_us":103,"threads_started":1}
I20260925 00:50:50.326085 14282 tablet_bootstrap.cc:492] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6: Bootstrap starting.
I20260925 00:50:50.327826 14282 tablet_bootstrap.cc:654] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:50.328433 14282 log.cc:824] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:50.329105 14281 raft_consensus.cc:348] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } }
I20260925 00:50:50.331073 14282 tablet_bootstrap.cc:492] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6: No bootstrap required, opened a new log
I20260925 00:50:50.331256 14282 ts_tablet_manager.cc:1397] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:50:50.334062 14180 tablet_service.cc:1467] Processing CreateTablet for tablet ae76d4417d8241cdb620e5ea12a9a721 (DEFAULT_TABLE table=test-table [id=8e7157bd71bd40a3b4241a95cce5b966]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:50.335157 14180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae76d4417d8241cdb620e5ea12a9a721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:50.336309 14282 raft_consensus.cc:348] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } }
I20260925 00:50:50.336587 14282 raft_consensus.cc:374] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:50.336743 14282 raft_consensus.cc:729] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038942e1595e47358520831c775e70b6, State: Initialized, Role: FOLLOWER
I20260925 00:50:50.337278 14282 consensus_queue.cc:260] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } }
I20260925 00:50:50.338392 14282 ts_tablet_manager.cc:1428] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 00:50:50.338833 13978 heartbeater.cc:503] Master 127.0.249.126:40937 was elected leader, sending a full tablet report...
I20260925 00:50:50.329383 14281 raft_consensus.cc:374] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:50.342267 14281 raft_consensus.cc:729] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d71c3c8aea4a0ea52111c1a33fd8f5, State: Initialized, Role: FOLLOWER
I20260925 00:50:50.343050 14281 consensus_queue.cc:260] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } }
I20260925 00:50:50.346683 14281 ts_tablet_manager.cc:1428] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260925 00:50:50.349691 14112 heartbeater.cc:503] Master 127.0.249.126:40937 was elected leader, sending a full tablet report...
I20260925 00:50:50.357769 14287 tablet_bootstrap.cc:492] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: Bootstrap starting.
I20260925 00:50:50.361177 14287 tablet_bootstrap.cc:654] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:50.362102 14287 log.cc:824] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:50.364740 14287 tablet_bootstrap.cc:492] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: No bootstrap required, opened a new log
I20260925 00:50:50.364934 14287 ts_tablet_manager.cc:1397] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 00:50:50.369062 14287 raft_consensus.cc:348] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } }
I20260925 00:50:50.369336 14287 raft_consensus.cc:374] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:50.369380 14287 raft_consensus.cc:729] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa949257c4824fe5b8cdc116fcd9dd1a, State: Initialized, Role: FOLLOWER
I20260925 00:50:50.381954 14287 consensus_queue.cc:260] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } }
I20260925 00:50:50.383365 14287 ts_tablet_manager.cc:1428] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: Time spent starting tablet: real 0.018s user 0.002s sys 0.004s
I20260925 00:50:50.383795 14246 heartbeater.cc:503] Master 127.0.249.126:40937 was elected leader, sending a full tablet report...
I20260925 00:50:50.426719 14285 raft_consensus.cc:482] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:50:50.426986 14285 raft_consensus.cc:504] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } }
I20260925 00:50:50.428450 14285 leader_election.cc:288] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa949257c4824fe5b8cdc116fcd9dd1a (127.0.249.67:42535), 75d71c3c8aea4a0ea52111c1a33fd8f5 (127.0.249.66:42405)
I20260925 00:50:50.441705 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.428777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:51435 (local address 127.0.249.67:42535)
0925 00:50:50.428910 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:50.428915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:50.433598 (+ 4683us) server_negotiation.cc:407] Connection header received
0925 00:50:50.433821 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:50.433825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:50.433881 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:50.433933 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:50.434386 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.435224 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.440154 (+ 4930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.440761 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.440859 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.440913 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:50.441484 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 00:50:50.441566 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:50:50.442395 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.428647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42535 (local address 127.0.249.65:51435)
0925 00:50:50.433488 (+ 4841us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:50.433505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:50.433686 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:50.434084 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:50.434087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:50.434104 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:50.434258 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.434264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.435405 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.435409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.436745 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.436755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.441035 (+ 4280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.441038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.441151 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.441303 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 00:50:50.442143 (+ 840us) client_negotiation.cc:240] Negotiation successful
0925 00:50:50.442234 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4719,"thread_start_us":122,"threads_started":1}
I20260925 00:50:50.443216 14200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae76d4417d8241cdb620e5ea12a9a721" candidate_uuid: "038942e1595e47358520831c775e70b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" is_pre_election: true
I20260925 00:50:50.443555 14200 raft_consensus.cc:2454] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038942e1595e47358520831c775e70b6 in term 0.
W20260925 00:50:50.448599 14113 tablet.cc:2369] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:50:50.449790 13867 leader_election.cc:302] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038942e1595e47358520831c775e70b6, aa949257c4824fe5b8cdc116fcd9dd1a; no voters:
I20260925 00:50:50.450119 14285 raft_consensus.cc:2781] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:50:50.450209 14285 raft_consensus.cc:482] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:50:50.450253 14285 raft_consensus.cc:3037] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:50.451566 14285 raft_consensus.cc:504] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } }
I20260925 00:50:50.451924 14285 leader_election.cc:288] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [CANDIDATE]: Term 1 election: Requested vote from peers aa949257c4824fe5b8cdc116fcd9dd1a (127.0.249.67:42535), 75d71c3c8aea4a0ea52111c1a33fd8f5 (127.0.249.66:42405)
I20260925 00:50:50.452512 14200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae76d4417d8241cdb620e5ea12a9a721" candidate_uuid: "038942e1595e47358520831c775e70b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a"
I20260925 00:50:50.452644 14200 raft_consensus.cc:3037] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:50.453950 14200 raft_consensus.cc:2454] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038942e1595e47358520831c775e70b6 in term 1.
I20260925 00:50:50.454362 13867 leader_election.cc:302] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038942e1595e47358520831c775e70b6, aa949257c4824fe5b8cdc116fcd9dd1a; no voters:
I20260925 00:50:50.454609 14285 raft_consensus.cc:2781] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:50:50.454849 14285 raft_consensus.cc:686] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [term 1 LEADER]: Becoming Leader. State: Replica: 038942e1595e47358520831c775e70b6, State: Running, Role: LEADER
I20260925 00:50:50.455577 14285 consensus_queue.cc:237] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } }
I20260925 00:50:50.459113 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.429347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:40611 (local address 127.0.249.66:42405)
0925 00:50:50.429472 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:50.429476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:50.437098 (+ 7622us) server_negotiation.cc:407] Connection header received
0925 00:50:50.437301 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:50.437304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:50.437354 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:50.437420 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:50.437805 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.438613 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.457720 (+ 19107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.458352 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.458451 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.458507 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:50.458889 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 00:50:50.458978 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 00:50:50.459115 13780 catalog_manager.cc:5495] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038942e1595e47358520831c775e70b6 (127.0.249.65). New cstate: current_term: 1 leader_uuid: "038942e1595e47358520831c775e70b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038942e1595e47358520831c775e70b6" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 34611 } health_report { overall_health: HEALTHY } } }
I20260925 00:50:50.459774 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.429024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:42405 (local address 127.0.249.65:40611)
0925 00:50:50.437005 (+ 7981us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:50.437020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:50.437178 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:50.437553 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:50.437556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:50.437572 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:50.437677 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.437683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.438771 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.438774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.440007 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:50.440017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.458615 (+ 18598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.458620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:50.458730 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.458754 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:50:50.459459 (+ 705us) client_negotiation.cc:240] Negotiation successful
0925 00:50:50.459634 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7914,"thread_start_us":72,"threads_started":1}
I20260925 00:50:50.460682 14065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae76d4417d8241cdb620e5ea12a9a721" candidate_uuid: "038942e1595e47358520831c775e70b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" is_pre_election: true
I20260925 00:50:50.460971 14065 raft_consensus.cc:2454] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038942e1595e47358520831c775e70b6 in term 0.
I20260925 00:50:50.461319 14062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae76d4417d8241cdb620e5ea12a9a721" candidate_uuid: "038942e1595e47358520831c775e70b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5"
I20260925 00:50:50.461438 14062 raft_consensus.cc:3037] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:50.462782 14062 raft_consensus.cc:2454] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038942e1595e47358520831c775e70b6 in term 1.
W20260925 00:50:50.468195 13979 tablet.cc:2369] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:50:50.476511 14247 tablet.cc:2369] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:50:50.508064 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.502904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.0.249.65:34611)
0925 00:50:50.503039 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:50.503043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:50.503058 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:50:50.503156 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:50.503159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:50.503246 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:50.503361 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:50:50.503776 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.504599 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.505888 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:50.506212 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:50.506321 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:50.506761 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:50:50.507334 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:50:50.507453 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:50.507843 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 00:50:50.507902 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:50:50.515308 14200 raft_consensus.cc:1261] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [term 1 FOLLOWER]: Refusing update from remote peer 038942e1595e47358520831c775e70b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:50.515965 14062 raft_consensus.cc:1261] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 1 FOLLOWER]: Refusing update from remote peer 038942e1595e47358520831c775e70b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:50.515941 14285 consensus_queue.cc:1035] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa949257c4824fe5b8cdc116fcd9dd1a" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:50:50.519488 14285 consensus_queue.cc:1035] T ae76d4417d8241cdb620e5ea12a9a721 P 038942e1595e47358520831c775e70b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d71c3c8aea4a0ea52111c1a33fd8f5" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:50:50.530995 14295 mvcc.cc:205] Tried to move back new op lower bound from 7333058357301485568 to 7333058357069701120. Current Snapshot: MvccSnapshot[applied={T|T < 7333058357301485568 or (T in {7333058357301485568})}]
I20260925 00:50:50.531206 14296 mvcc.cc:205] Tried to move back new op lower bound from 7333058357301485568 to 7333058357069701120. Current Snapshot: MvccSnapshot[applied={T|T < 7333058357301485568}]
I20260925 00:50:50.532043 13779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:50:50.532213 13779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:50:50.534698 13779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7964dab02b76435e81ae927203016511: Sending DeleteTablet for 3 replicas of tablet ae76d4417d8241cdb620e5ea12a9a721
I20260925 00:50:50.535586 14180 tablet_service.cc:1514] Processing DeleteTablet for tablet ae76d4417d8241cdb620e5ea12a9a721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:50 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:47424
I20260925 00:50:50.536320 13912 tablet_service.cc:1514] Processing DeleteTablet for tablet ae76d4417d8241cdb620e5ea12a9a721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:50 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:51560
I20260925 00:50:50.537415 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 13818
I20260925 00:50:50.541447 14034 tablet_service.cc:1514] Processing DeleteTablet for tablet ae76d4417d8241cdb620e5ea12a9a721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:50 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:43616
I20260925 00:50:50.541898 14305 tablet_replica.cc:330] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: stopping tablet replica
I20260925 00:50:50.542238 14305 raft_consensus.cc:2229] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:50:50.542485 14305 pending_rounds.cc:62] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Trying to abort 2 pending ops.
I20260925 00:50:50.542563 14305 pending_rounds.cc:68] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333058357301485568 op_type: WRITE_OP write_request { tablet_id: "ae76d4417d8241cdb620e5ea12a9a721" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\226\215\327\325\006\"8\n\ntest-admin\022*\n 8e7157bd71bd40a3b4241a95cce5b966\020\001\030\001 \001(\001" signature: "<redacted>""cfW\265\207\004\020z\005u|!\323@|(Y\375)s\2503\343\200(\262\031\356\000\\\332\302Z\340\267\014j\210Yamr\251^\370Q\021\304\017\274Lvt\224\231]\231\310\263\303\301\366\247\303" signing_key_seq_num: 0 } } request_id { client_id: "a876c15e69804f42ac5ba2fef293b583" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260925 00:50:50.543731 14305 pending_rounds.cc:68] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333058357069701120 op_type: NO_OP noop_request { }
I20260925 00:50:50.543843 14305 raft_consensus.cc:2866] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260925 00:50:50.543937 14305 raft_consensus.cc:2258] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:50:50.545481 14305 ts_tablet_manager.cc:1905] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:50:50.546430 14002 connection.cc:535] server connection from 127.0.249.65:40611 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 00:50:50.546875 14303 tablet_replica.cc:330] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: stopping tablet replica
I20260925 00:50:50.547217 14303 raft_consensus.cc:2229] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:50:50.547549 14303 raft_consensus.cc:2258] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:50:50.548471 14305 ts_tablet_manager.cc:1918] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:50:50.548573 14305 log.cc:1196] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/ae76d4417d8241cdb620e5ea12a9a721
I20260925 00:50:50.548892 14305 ts_tablet_manager.cc:1939] T ae76d4417d8241cdb620e5ea12a9a721 P 75d71c3c8aea4a0ea52111c1a33fd8f5: Deleting consensus metadata
I20260925 00:50:50.549177 14303 ts_tablet_manager.cc:1905] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:50:50.550249 13766 catalog_manager.cc:4841] TS 75d71c3c8aea4a0ea52111c1a33fd8f5 (127.0.249.66:42405): tablet ae76d4417d8241cdb620e5ea12a9a721 (table test-table [id=8e7157bd71bd40a3b4241a95cce5b966]) successfully deleted
I20260925 00:50:50.551703 14303 ts_tablet_manager.cc:1918] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:50:50.551810 14303 log.cc:1196] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/ae76d4417d8241cdb620e5ea12a9a721
I20260925 00:50:50.552138 14303 ts_tablet_manager.cc:1939] T ae76d4417d8241cdb620e5ea12a9a721 P aa949257c4824fe5b8cdc116fcd9dd1a: Deleting consensus metadata
I20260925 00:50:50.553349 13766 catalog_manager.cc:4841] TS aa949257c4824fe5b8cdc116fcd9dd1a (127.0.249.67:42535): tablet ae76d4417d8241cdb620e5ea12a9a721 (table test-table [id=8e7157bd71bd40a3b4241a95cce5b966]) successfully deleted
W20260925 00:50:50.559149 13764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:34611 (error 108)
I20260925 00:50:50.562870 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 13982
I20260925 00:50:50.564414 14276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:50.563217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:34611 (local address 127.0.0.1:49178)
0925 00:50:50.564216 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:50.564300 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":899}
W20260925 00:50:50.564996 13764 catalog_manager.cc:4584] TS 038942e1595e47358520831c775e70b6 (127.0.249.65:34611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae76d4417d8241cdb620e5ea12a9a721: Network error: Client connection negotiation failed: client connection to 127.0.249.65:34611: connect: Connection refused (error 111)
I20260925 00:50:50.577775 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 14115
I20260925 00:50:50.584493 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 13747
I20260925 00:50:50.591152 997 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 00:50:50.936055 997 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 00:50:50.936177 997 mini_ranger.cc:66] Stopping Ranger...
I20260925 00:50:51.280889 997 mini_ranger.cc:68] Stopped Ranger
2026-09-25 00:50:51.281 UTC [12814] LOG: received smart shutdown request
2026-09-25 00:50:51.284 UTC [12814] LOG: background worker "logical replication launcher" (PID 12823) exited with exit code 1
2026-09-25 00:50:51.285 UTC [12818] LOG: shutting down
2026-09-25 00:50:51.450 UTC [12814] LOG: database system is shut down
2026-09-25T00:50:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46715 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:50:51 dist-test-slave-05m0 krb5kdc[14322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:50:51 dist-test-slave-05m0 krb5kdc[14322](info): set up 2 sockets
Sep 25 00:50:51 dist-test-slave-05m0 krb5kdc[14322](info): commencing operation
krb5kdc: starting...
W20260925 00:50:52.601799 997 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:50:52 dist-test-slave-05m0 krb5kdc[14322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:50:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:50:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:50:52.721204 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:44007
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:44007,127.0.249.125:42551,127.0.249.124:37273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:52.829223 14335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:52.829522 14335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:52.829605 14335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:52.833619 14335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:52.833703 14335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:52.833743 14335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:52.833801 14335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:52.833861 14335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:52.838996 14335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:44007,127.0.249.125:42551,127.0.249.124:37273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:52.839998 14335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:52.840984 14335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:52.849382 14343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:52.849371 14340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:52.849371 14341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:52.849589 14335 server_base.cc:1034] running on GCE node
I20260925 00:50:52.850208 14335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:52.851240 14335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:52.852425 14335 hybrid_clock.cc:648] HybridClock initialized: now 1790297452852410 us; error 25 us; skew 500 ppm
Sep 25 00:50:52 dist-test-slave-05m0 krb5kdc[14322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:52.855168 14335 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:50:52.856024 14335 webserver.cc:460] Webserver started at http://127.0.249.126:43375/ using document root <none> and password file <none>
I20260925 00:50:52.856588 14335 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:52.856674 14335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:52.856904 14335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:50:52.858919 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "bf1596273a1e4f27b3d496eb4a615a52"
format_stamp: "Formatted at 2026-09-25 00:50:52 on dist-test-slave-05m0"
server_key: "5e169daf5326981e3a1fa7e76f84a2c3"
server_key_iv: "6044f64bac906dae8dd691c567be78e4"
server_key_version: "encryptionkey@0"
I20260925 00:50:52.859467 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "bf1596273a1e4f27b3d496eb4a615a52"
format_stamp: "Formatted at 2026-09-25 00:50:52 on dist-test-slave-05m0"
server_key: "5e169daf5326981e3a1fa7e76f84a2c3"
server_key_iv: "6044f64bac906dae8dd691c567be78e4"
server_key_version: "encryptionkey@0"
I20260925 00:50:52.863130 14335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:50:52.865357 14350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:52.866542 14335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:50:52.866686 14335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "bf1596273a1e4f27b3d496eb4a615a52"
format_stamp: "Formatted at 2026-09-25 00:50:52 on dist-test-slave-05m0"
server_key: "5e169daf5326981e3a1fa7e76f84a2c3"
server_key_iv: "6044f64bac906dae8dd691c567be78e4"
server_key_version: "encryptionkey@0"
I20260925 00:50:52.866804 14335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:52.885306 14335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:52.888577 14335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:52.888778 14335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:52.895648 14335 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:44007
I20260925 00:50:52.895665 14402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:44007 every 8 connection(s)
I20260925 00:50:52.896541 14335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:50:52.897044 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 14335
I20260925 00:50:52.897178 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:50:52.897461 997 external_mini_cluster.cc:1428] Setting key 743cb785790cb23410358dcd45ae88e9
I20260925 00:50:52.899294 14403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:52.902998 14403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.249.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.125@KRBTEST.COM" created.
Sep 25 00:50:52 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:52.919514 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:52.905101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44007 (local address 127.0.0.1:55168)
0925 00:50:52.905649 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:52.905689 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:52.906550 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:52.907475 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:52.907484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:52.907959 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:52.908354 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:52.908376 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:52.910314 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:52.910320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:52.911619 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:52.911630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:52.912319 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:52.912327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:52.912483 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:52.913062 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:52.913086 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:52.914712 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:52.916646 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:52.916655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:52.916664 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:52.916903 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:52.917216 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:52.917221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:52.917223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:52.917334 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:52.917895 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:52.917901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:52.918037 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 00:50:52.918195 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 00:50:52.918568 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":105,"thread_start_us":116,"threads_started":1}
I20260925 00:50:52.919516 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:52.905872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.0.249.126:44007)
0925 00:50:52.906128 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:52.906137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:52.906172 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:50:52.906964 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:52.906974 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:52.907262 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:52.907440 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:52.908564 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:52.910183 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:52.911744 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:52.912105 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:52.912258 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:52.914889 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:52.914895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:52.914904 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:52.914922 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:52.916522 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:52.917012 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:52.917018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:52.917019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:52.917124 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:52.917441 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:52.917446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:52.917448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:52.917783 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:52.917922 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:52.918279 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 00:50:52.918519 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260925 00:50:52.921808 14403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:50:52.922464 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:52.922239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42551 (local address 127.0.0.1:34850)
0925 00:50:52.922344 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:52.922408 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 00:50:52.922690 14351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab.
Entry for principal kudu/127.0.249.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab.
I20260925 00:50:52.923569 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:52.923364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42551 (local address 127.0.0.1:34864)
0925 00:50:52.923467 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:52.923512 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 00:50:52.923777 14403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.125:42551: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
I20260925 00:50:52.924054 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.125:42551
--webserver_interface=127.0.249.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab
--principal=kudu/127.0.249.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:44007,127.0.249.125:42551,127.0.249.124:37273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:50:52.973006 14403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } attempt: 1
I20260925 00:50:52.973930 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:52.973637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42551 (local address 127.0.0.1:34874)
0925 00:50:52.973784 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:52.973854 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 00:50:52.974684 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:52.974468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42551 (local address 127.0.0.1:34884)
0925 00:50:52.974579 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:52.974633 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 00:50:52.974874 14403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.125:42551: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
W20260925 00:50:53.031178 14410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:53.031414 14410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:53.031452 14410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:53.035394 14410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:53.035445 14410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:53.035465 14410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:53.035481 14410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:53.035495 14410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:53.040530 14410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:44007,127.0.249.125:42551,127.0.249.124:37273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab
--principal=kudu/127.0.249.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.125:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.249.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:53.041481 14410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:53.042461 14410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:50:53.043022 14403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } attempt: 2
I20260925 00:50:53.043977 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.043612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42551 (local address 127.0.0.1:34892)
0925 00:50:53.043828 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.043897 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 00:50:53.044806 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.044575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42551 (local address 127.0.0.1:34900)
0925 00:50:53.044677 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.044725 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 00:50:53.045011 14403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.125:42551: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42551: connect: Connection refused (error 111)
W20260925 00:50:53.051129 14415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:53.051151 14418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:53.051383 14410 server_base.cc:1034] running on GCE node
W20260925 00:50:53.051129 14416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:53.052024 14410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:53.053066 14410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:53.054224 14410 hybrid_clock.cc:648] HybridClock initialized: now 1790297453054206 us; error 25 us; skew 500 ppm
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:53.057039 14410 init.cc:377] Logged in from keytab as kudu/127.0.249.125@KRBTEST.COM (short username kudu)
I20260925 00:50:53.057905 14410 webserver.cc:460] Webserver started at http://127.0.249.125:37643/ using document root <none> and password file <none>
I20260925 00:50:53.058434 14410 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:53.058517 14410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:53.058758 14410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:50:53.060671 14410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/data/instance:
uuid: "addf2a47fcc54adc93efab63e90b36c1"
format_stamp: "Formatted at 2026-09-25 00:50:53 on dist-test-slave-05m0"
server_key: "4f3e4714228c7568879eb40fbca56902"
server_key_iv: "c2d749fb0ce80196361a3952a0387c8c"
server_key_version: "encryptionkey@0"
I20260925 00:50:53.061275 14410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/wal/instance:
uuid: "addf2a47fcc54adc93efab63e90b36c1"
format_stamp: "Formatted at 2026-09-25 00:50:53 on dist-test-slave-05m0"
server_key: "4f3e4714228c7568879eb40fbca56902"
server_key_iv: "c2d749fb0ce80196361a3952a0387c8c"
server_key_version: "encryptionkey@0"
I20260925 00:50:53.065042 14410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:50:53.067380 14425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:53.068502 14410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:50:53.068632 14410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/wal
uuid: "addf2a47fcc54adc93efab63e90b36c1"
format_stamp: "Formatted at 2026-09-25 00:50:53 on dist-test-slave-05m0"
server_key: "4f3e4714228c7568879eb40fbca56902"
server_key_iv: "c2d749fb0ce80196361a3952a0387c8c"
server_key_version: "encryptionkey@0"
I20260925 00:50:53.068737 14410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:53.079525 14410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:53.082119 14410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:53.082297 14410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:53.088919 14410 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.125:42551
I20260925 00:50:53.088943 14477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.125:42551 every 8 connection(s)
I20260925 00:50:53.089854 14410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
I20260925 00:50:53.092370 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:53.095882 14478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:50:53.099587 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 14410
I20260925 00:50:53.099717 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-1/wal/instance
I20260925 00:50:53.099980 997 external_mini_cluster.cc:1428] Setting key 65146d3e08a65f42adb49e25968f4328
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.124@KRBTEST.COM" created.
I20260925 00:50:53.109683 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.097638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.0.249.126:44007)
0925 00:50:53.097775 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.097778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.098494 (+ 716us) server_negotiation.cc:407] Connection header received
0925 00:50:53.099359 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.099362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.099423 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.099533 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.100738 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.101250 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.102398 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.102658 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.102788 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.105475 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.105478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.105481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.105502 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.107609 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.108111 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.108116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.108118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.108206 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.108538 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.108542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.108544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.108723 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.108831 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.109339 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.109493 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:50:53.110435 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.097927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44007 (local address 127.0.0.1:55182)
0925 00:50:53.098333 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.098392 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:53.099150 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:53.099714 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:53.099721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:53.100133 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:53.100584 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.100597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.101373 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.101378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.102271 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.102281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.102819 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.102823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.102991 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.103578 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:53.103594 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:53.105276 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:53.107728 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.107734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.107748 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.108006 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.108306 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.108309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.108311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.108443 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.108833 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:53.108840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:53.109083 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 00:50:53.109271 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 00:50:53.109641 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260925 00:50:53.111820 14478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
I20260925 00:50:53.121948 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.112316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42551 (local address 127.0.0.1:34916)
0925 00:50:53.112643 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.112668 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:53.112754 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:53.113153 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:53.113156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:53.113312 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:53.113438 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.113445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.114363 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.114369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.114894 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.114901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.115468 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.115472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.115557 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.116005 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:53.116018 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:53.118561 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:53.120471 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.120475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.120478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.120661 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.120929 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.120933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.120935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.121012 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.121441 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:53.121444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:53.121521 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 00:50:53.121636 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 00:50:53.121757 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275}
I20260925 00:50:53.122007 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.112388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.0.249.125:42551)
0925 00:50:53.112635 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.112649 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.112738 (+ 89us) server_negotiation.cc:407] Connection header received
0925 00:50:53.112845 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.112850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.113043 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.113193 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.113543 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.114248 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.115010 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.115361 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.115475 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.118784 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.118792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.118801 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.118820 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.120339 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.120761 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.120766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.120768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.120840 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.121095 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.121100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.121101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.121312 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.121461 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.121719 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.121832 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab.
Entry for principal kudu/127.0.249.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab.
I20260925 00:50:53.123704 14478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:50:53.123871 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.124:37273
--webserver_interface=127.0.249.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab
--principal=kudu/127.0.249.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:44007,127.0.249.125:42551,127.0.249.124:37273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:50:53.124368 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.124119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47652)
0925 00:50:53.124226 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.124299 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 00:50:53.124562 14429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
I20260925 00:50:53.125378 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.125208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47660)
0925 00:50:53.125290 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.125332 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 00:50:53.125660 14478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:37273: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
I20260925 00:50:53.136206 14403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } attempt: 3
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
I20260925 00:50:53.145486 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.136705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42551 (local address 127.0.0.1:34926)
0925 00:50:53.136837 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.136853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:53.136955 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:53.137215 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:53.137217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:53.137485 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:53.137597 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.137603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.139120 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.139123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.139665 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.139672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.140293 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.140295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.140374 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.140898 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:53.140910 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:53.142325 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:53.144048 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.144053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.144055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.144290 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.144549 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.144552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.144554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.144644 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.144968 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:53.144970 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:53.145068 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 00:50:53.145174 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 00:50:53.145301 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 00:50:53.145720 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.136747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.0.249.125:42551)
0925 00:50:53.136862 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.136867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.136906 (+ 39us) server_negotiation.cc:407] Connection header received
0925 00:50:53.137053 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.137056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.137101 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.137187 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.137702 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.139008 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.139767 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.140185 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.140284 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.142453 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.142456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.142459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.142479 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.143920 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.144397 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.144399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.144401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.144469 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.144727 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.144729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.144731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.144881 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.144952 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.145299 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.145478 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:50:53.146423 14403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:50:53.147110 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.146873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47668)
0925 00:50:53.146992 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.147046 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 00:50:53.147855 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.147597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47680)
0925 00:50:53.147766 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.147809 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260925 00:50:53.148032 14403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:37273: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
I20260925 00:50:53.174896 14478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } attempt: 1
I20260925 00:50:53.175769 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.175446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47682)
0925 00:50:53.175621 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.175684 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 00:50:53.176463 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.176269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47696)
0925 00:50:53.176369 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.176412 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 00:50:53.176636 14478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:37273: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
I20260925 00:50:53.179185 14403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } attempt: 1
I20260925 00:50:53.179876 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.179610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47706)
0925 00:50:53.179760 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.179810 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 00:50:53.180518 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.180339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47720)
0925 00:50:53.180431 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.180471 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 00:50:53.180656 14403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:37273: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
W20260925 00:50:53.235718 14485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:53.235973 14485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:53.236012 14485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:53.239957 14485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:53.240016 14485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:53.240036 14485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:53.240082 14485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:53.240118 14485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:53.244825 14478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } attempt: 2
I20260925 00:50:53.245757 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.245426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47730)
0925 00:50:53.245609 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.245677 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 00:50:53.246194 14485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:44007,127.0.249.125:42551,127.0.249.124:37273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab
--principal=kudu/127.0.249.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.124:37273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.249.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:53.246842 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.246600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47742)
0925 00:50:53.246737 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.246785 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 00:50:53.247004 14478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:37273: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
I20260925 00:50:53.247429 14485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:53.248448 14485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:50:53.255937 14403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } attempt: 2
I20260925 00:50:53.256927 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.256439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47744)
0925 00:50:53.256574 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.256765 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 00:50:53.256868 14491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:53.256888 14490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:53.256873 14493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:53.257853 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.257592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47748)
0925 00:50:53.257737 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.257801 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 00:50:53.258127 14403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:37273: Network error: Client connection negotiation failed: client connection to 127.0.249.124:37273: connect: Connection refused (error 111)
I20260925 00:50:53.258172 14485 server_base.cc:1034] running on GCE node
I20260925 00:50:53.258700 14485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:53.259894 14485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:53.261081 14485 hybrid_clock.cc:648] HybridClock initialized: now 1790297453261066 us; error 30 us; skew 500 ppm
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:53.264043 14485 init.cc:377] Logged in from keytab as kudu/127.0.249.124@KRBTEST.COM (short username kudu)
I20260925 00:50:53.264968 14485 webserver.cc:460] Webserver started at http://127.0.249.124:33427/ using document root <none> and password file <none>
I20260925 00:50:53.265578 14485 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:53.265667 14485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:53.265897 14485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:50:53.267863 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/data/instance:
uuid: "4497702ece0b4426abfafb3bd508f695"
format_stamp: "Formatted at 2026-09-25 00:50:53 on dist-test-slave-05m0"
server_key: "d96f7e67083d5e79990a197c5c5ea682"
server_key_iv: "0e2445253520ac327adcf3ec1b678f20"
server_key_version: "encryptionkey@0"
I20260925 00:50:53.268406 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/wal/instance:
uuid: "4497702ece0b4426abfafb3bd508f695"
format_stamp: "Formatted at 2026-09-25 00:50:53 on dist-test-slave-05m0"
server_key: "d96f7e67083d5e79990a197c5c5ea682"
server_key_iv: "0e2445253520ac327adcf3ec1b678f20"
server_key_version: "encryptionkey@0"
I20260925 00:50:53.272374 14485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 00:50:53.274734 14500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:53.275928 14485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:50:53.276067 14485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/wal
uuid: "4497702ece0b4426abfafb3bd508f695"
format_stamp: "Formatted at 2026-09-25 00:50:53 on dist-test-slave-05m0"
server_key: "d96f7e67083d5e79990a197c5c5ea682"
server_key_iv: "0e2445253520ac327adcf3ec1b678f20"
server_key_version: "encryptionkey@0"
I20260925 00:50:53.276175 14485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:53.286193 14485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:53.289556 14485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:53.289757 14485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:53.296789 14552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.124:37273 every 8 connection(s)
I20260925 00:50:53.296794 14485 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.124:37273
I20260925 00:50:53.297875 14485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
I20260925 00:50:53.299783 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 14485
I20260925 00:50:53.299922 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790297292685301-997-0/minicluster-data/master-2/wal/instance
I20260925 00:50:53.300185 997 external_mini_cluster.cc:1428] Setting key f345544d22177453b320335676748ca8
I20260925 00:50:53.300750 14553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:53.305617 14553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:53.315477 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.301621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.0.249.124:37273)
0925 00:50:53.302018 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.302036 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.302077 (+ 41us) server_negotiation.cc:407] Connection header received
0925 00:50:53.302881 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.302904 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.303326 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.303693 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.304317 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.305825 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.306907 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.307451 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.307623 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.310485 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.310490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.310499 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.310526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.312457 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.312944 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.312949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.312954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.313068 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.313459 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.313463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.313464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.313911 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.314142 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.314463 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.314754 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260925 00:50:53.316398 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.307503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.0.249.126:44007)
0925 00:50:53.307644 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.307649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.307929 (+ 280us) server_negotiation.cc:407] Connection header received
0925 00:50:53.308101 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.308104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.308145 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.308228 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.308882 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.309363 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.310382 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.310621 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.310697 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.313149 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.313152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.313154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.313176 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.314708 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.315153 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.315158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.315160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.315233 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.315506 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.315511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.315512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.315715 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.315808 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.316142 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.316251 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:50:53.316402 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.307423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44007 (local address 127.0.0.1:55198)
0925 00:50:53.307809 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.307851 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:53.307980 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:53.308277 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:53.308282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:53.308633 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:53.308757 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.308763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.309516 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.309519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.310256 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.310266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.310736 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.310741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.310882 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.311331 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:53.311352 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:53.313015 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:53.314812 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.314822 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.314827 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.315054 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.315328 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.315332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.315334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.315417 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.315807 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:53.315814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:53.315930 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 00:50:53.316063 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 00:50:53.316236 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":118,"threads_started":1}
I20260925 00:50:53.317319 14553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:53.327713 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.317946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.0.249.125:42551)
0925 00:50:53.318077 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.318081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.318094 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:50:53.318137 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.318140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.318182 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.318267 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.318756 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.319512 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.320500 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.320856 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.320959 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.323499 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.323503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.323506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.323532 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.325866 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.326356 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.326363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.326368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.326453 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.326763 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.326766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.326768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.327004 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.327077 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.327431 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.327545 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:50:53.328142 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.317824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42551 (local address 127.0.0.1:34936)
0925 00:50:53.317921 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.317936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:53.318003 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:53.318309 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:53.318313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:53.318494 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:53.318621 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.318640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.319624 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.319627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.320381 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.320389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.320994 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.320998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.321204 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.321724 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:53.321743 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:53.323338 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:53.325992 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.325995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.326001 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.326240 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.326575 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.326578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.326579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.326660 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.327152 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:53.327156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:53.327246 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 00:50:53.327866 (+ 620us) client_negotiation.cc:240] Negotiation successful
0925 00:50:53.327997 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 00:50:53.328141 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.318183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.0.249.126:44007)
0925 00:50:53.318335 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.318341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.318481 (+ 140us) server_negotiation.cc:407] Connection header received
0925 00:50:53.318567 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.318571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.318619 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.318710 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.319416 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.320118 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.321137 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.321538 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.321639 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.324543 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.324549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.324553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.324587 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.326295 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.326764 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.326767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.326769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.326841 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.327144 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.327147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.327148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.327359 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.327545 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.327896 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.327997 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 00:50:53.334052 14553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:50:53.338173 14478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } attempt: 3
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
I20260925 00:50:53.345250 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.333560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.0.249.125:42551)
0925 00:50:53.333740 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.333744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.335208 (+ 1464us) server_negotiation.cc:407] Connection header received
0925 00:50:53.335256 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.335259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.335306 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.335381 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.336380 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.337121 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.338069 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.338368 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.338460 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.341303 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.341306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.341309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.341339 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.343245 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.343740 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.343744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.343746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.343821 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.344124 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.344129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.344134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.344327 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.344556 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.344938 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.345054 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 00:50:53.345605 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.334577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.0.249.124:37273)
0925 00:50:53.334863 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.334867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.334878 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:50:53.334907 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.334910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.334945 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.335019 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.335551 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.336702 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.337542 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.337797 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.337896 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.341265 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.341271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.341277 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.341306 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.343005 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.343474 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.343480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.343484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.343574 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.343896 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.343899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.343901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.344104 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.344189 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.345273 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.345473 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":224}
I20260925 00:50:53.345777 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.334525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47774)
0925 00:50:53.334617 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.334632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:53.334704 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:53.335109 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:53.335111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:53.335308 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:53.335393 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.335399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.336815 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.336822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.337381 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.337440 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.337905 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.337908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.338010 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.338748 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:53.338767 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:53.341120 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:53.343126 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.343129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.343131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.343371 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.343680 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.343684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.343687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.343778 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.344221 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:53.344224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:53.344296 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 00:50:53.344667 (+ 371us) client_negotiation.cc:240] Negotiation successful
0925 00:50:53.344797 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":19}
I20260925 00:50:53.348649 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:50:53.349695 14553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695: Bootstrap starting.
I20260925 00:50:53.350244 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.338526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47780)
0925 00:50:53.338629 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.338642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:53.338714 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:53.339278 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:53.339281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:53.339449 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:53.339515 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.339520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.340584 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.340586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.341192 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.341205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.341857 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.341862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.341961 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.342519 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:53.342535 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:53.344057 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:53.346292 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.346295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.346297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.346577 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.346984 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.346987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.346989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.347076 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.349807 (+ 2731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:53.349810 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:53.349900 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 00:50:53.349979 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 00:50:53.350102 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 00:50:53.350821 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.338742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.0.249.124:37273)
0925 00:50:53.339055 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.339064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.339077 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:50:53.339110 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.339116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.339157 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.339265 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.339644 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.340476 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.341325 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.341706 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.341847 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.344183 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.344186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.344189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.344211 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.346198 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.346717 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.346730 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.346731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.346816 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.349475 (+ 2659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.349478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.349480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.349667 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.350491 (+ 824us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.350561 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.350661 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260925 00:50:53.356411 14478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1: Bootstrap starting.
I20260925 00:50:53.357297 14403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } attempt: 3
I20260925 00:50:53.352643 14553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:53.358412 14553 log.cc:824] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:53.359014 14478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:53.359894 14478 log.cc:824] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:50:53 dist-test-slave-05m0 krb5kdc[14322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
I20260925 00:50:53.364711 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.351450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.0.249.124:37273)
0925 00:50:53.351546 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.351549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.351940 (+ 391us) server_negotiation.cc:407] Connection header received
0925 00:50:53.352039 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.352043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.352085 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.352164 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.352802 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.353551 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.355029 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.355342 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.355399 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.360749 (+ 5350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.360755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.360758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.360787 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.362649 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.363231 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.363236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.363239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.363346 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.363680 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.363686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.363690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.363918 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.364026 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.364436 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.364550 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 00:50:53.365782 14478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1: No bootstrap required, opened a new log
I20260925 00:50:53.366750 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.351078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.0.249.125:42551)
0925 00:50:53.351181 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.351185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.351198 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:50:53.351230 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.351233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.351272 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.351345 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.352226 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.352865 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.354088 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.354407 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.354503 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.362379 (+ 7876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.362383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.362388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.362417 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.364621 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.365258 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.365261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.365262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.365328 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.365675 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.365680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.365682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.365909 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.366028 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.366512 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.366619 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:50:53.366922 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.357717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:37273 (local address 127.0.0.1:47790)
0925 00:50:53.357886 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:53.357899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:53.357966 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:53.358329 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:53.358332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:53.358511 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:53.358603 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.358610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.359592 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.359596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.360354 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:53.360364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.360941 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.360945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:53.361036 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.361615 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:53.361629 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:53.363189 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:53.365081 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.365084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.365085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.365310 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.365924 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:53.365928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:53.365929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:53.366017 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:53.366453 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:53.366456 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:53.366538 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 00:50:53.366632 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 00:50:53.366764 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 00:50:53.367492 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.357778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.0.249.124:37273)
0925 00:50:53.358127 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.358131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.358142 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:50:53.358173 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.358175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.358212 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.358287 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.358729 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.359435 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.360471 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.360802 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.361233 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.363345 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.363349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.363354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.363381 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.364957 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.365451 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.365456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.365457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.365538 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.366133 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.366136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.366138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.366323 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.366410 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.367229 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.367341 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260925 00:50:53.367987 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:53.350903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.249.126:44007)
0925 00:50:53.352984 (+ 2081us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:53.352987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:53.352998 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:50:53.353032 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:53.353034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:53.353070 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:53.353162 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:53.354009 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.354676 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.355777 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:53.356078 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:53.356178 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:53.360697 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:53.360701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:53.360704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:53.360731 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:53.362889 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.363384 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.363387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.363389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.363461 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:53.363820 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:53.363823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:53.363825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:53.364055 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:53.364185 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:53.367738 (+ 3553us) server_negotiation.cc:299] Negotiation successful
0925 00:50:53.367849 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 00:50:53.369809 14478 raft_consensus.cc:348] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } }
I20260925 00:50:53.370088 14478 raft_consensus.cc:374] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:53.370141 14478 raft_consensus.cc:729] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: addf2a47fcc54adc93efab63e90b36c1, State: Initialized, Role: FOLLOWER
I20260925 00:50:53.370232 14553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695: No bootstrap required, opened a new log
I20260925 00:50:53.370602 14403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52: Bootstrap starting.
I20260925 00:50:53.370631 14478 consensus_queue.cc:260] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } }
I20260925 00:50:53.371127 14568 sys_catalog.cc:455] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.371434 14568 sys_catalog.cc:458] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:50:53.372085 14478 sys_catalog.cc:564] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:50:53.374300 14403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:53.374280 14553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } }
I20260925 00:50:53.374527 14553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:53.374594 14553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4497702ece0b4426abfafb3bd508f695, State: Initialized, Role: FOLLOWER
I20260925 00:50:53.375031 14553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } }
I20260925 00:50:53.375346 14403 log.cc:824] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:53.376297 14553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 00:50:53.378574 14579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:50:53.378717 14579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:50:53.379247 14403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52: No bootstrap required, opened a new log
W20260925 00:50:53.381517 14593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:50:53.381597 14593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:50:53.379635 14485 master_runner.cc:186] Error getting master registration for 127.0.249.126:44007: OK, instance_id { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" instance_seqno: 1790297452893732 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 00:50:53.375505 14410 master_runner.cc:186] Error getting master registration for 127.0.249.126:44007: OK, instance_id { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" instance_seqno: 1790297452893732 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 00:50:53.382879 14485 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:50:53.382933 14410 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:50:53.383177 14580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.383318 14580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:50:53.383941 14403 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } }
I20260925 00:50:53.384238 14403 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:53.384308 14403 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1596273a1e4f27b3d496eb4a615a52, State: Initialized, Role: FOLLOWER
I20260925 00:50:53.384987 14403 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } }
I20260925 00:50:53.387609 14403 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 00:50:53.390194 14356 tablet.cc:2369] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:50:53.393039 14606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:50:53.393113 14606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:50:53.396221 14595 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.396335 14595 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:50:53.397689 14335 master_runner.cc:263] No leader master found from master bf1596273a1e4f27b3d496eb4a615a52
I20260925 00:50:53.397751 14335 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:50:53.451787 14568 raft_consensus.cc:482] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:50:53.451979 14568 raft_consensus.cc:504] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } }
I20260925 00:50:53.453061 14568 leader_election.cc:288] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf1596273a1e4f27b3d496eb4a615a52 (127.0.249.126:44007), 4497702ece0b4426abfafb3bd508f695 (127.0.249.124:37273)
I20260925 00:50:53.454779 14377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "addf2a47fcc54adc93efab63e90b36c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1596273a1e4f27b3d496eb4a615a52" is_pre_election: true
I20260925 00:50:53.455127 14377 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate addf2a47fcc54adc93efab63e90b36c1 in term 0.
I20260925 00:50:53.455971 14428 leader_election.cc:302] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: addf2a47fcc54adc93efab63e90b36c1, bf1596273a1e4f27b3d496eb4a615a52; no voters:
I20260925 00:50:53.456588 14568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:50:53.459923 14527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "addf2a47fcc54adc93efab63e90b36c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4497702ece0b4426abfafb3bd508f695" is_pre_election: true
I20260925 00:50:53.460213 14527 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate addf2a47fcc54adc93efab63e90b36c1 in term 0.
I20260925 00:50:53.456872 14568 raft_consensus.cc:482] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:50:53.460602 14568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:53.462972 14568 raft_consensus.cc:504] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } }
I20260925 00:50:53.463330 14568 leader_election.cc:288] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [CANDIDATE]: Term 1 election: Requested vote from peers bf1596273a1e4f27b3d496eb4a615a52 (127.0.249.126:44007), 4497702ece0b4426abfafb3bd508f695 (127.0.249.124:37273)
I20260925 00:50:53.463758 14377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "addf2a47fcc54adc93efab63e90b36c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1596273a1e4f27b3d496eb4a615a52"
I20260925 00:50:53.463872 14377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:53.465157 14377 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate addf2a47fcc54adc93efab63e90b36c1 in term 1.
I20260925 00:50:53.465545 14428 leader_election.cc:302] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: addf2a47fcc54adc93efab63e90b36c1, bf1596273a1e4f27b3d496eb4a615a52; no voters:
I20260925 00:50:53.466028 14527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "addf2a47fcc54adc93efab63e90b36c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4497702ece0b4426abfafb3bd508f695"
I20260925 00:50:53.466144 14527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:53.467347 14527 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate addf2a47fcc54adc93efab63e90b36c1 in term 1.
I20260925 00:50:53.468814 14568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:50:53.469048 14568 raft_consensus.cc:686] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [term 1 LEADER]: Becoming Leader. State: Replica: addf2a47fcc54adc93efab63e90b36c1, State: Running, Role: LEADER
I20260925 00:50:53.469494 14568 consensus_queue.cc:237] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } }
I20260925 00:50:53.471534 14568 sys_catalog.cc:455] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader addf2a47fcc54adc93efab63e90b36c1. Latest consensus state: current_term: 1 leader_uuid: "addf2a47fcc54adc93efab63e90b36c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.471748 14568 sys_catalog.cc:458] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:53.472996 14610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:50:53.474125 14610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:50:53.482393 14527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [term 1 FOLLOWER]: Refusing update from remote peer addf2a47fcc54adc93efab63e90b36c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:53.483094 14608 consensus_queue.cc:1035] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:50:53.486819 14580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader addf2a47fcc54adc93efab63e90b36c1. Latest consensus state: current_term: 1 leader_uuid: "addf2a47fcc54adc93efab63e90b36c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.486948 14580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:50:53.487931 14377 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [term 1 FOLLOWER]: Refusing update from remote peer addf2a47fcc54adc93efab63e90b36c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:53.488570 14608 consensus_queue.cc:1035] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:50:53.492606 14595 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader addf2a47fcc54adc93efab63e90b36c1. Latest consensus state: current_term: 1 leader_uuid: "addf2a47fcc54adc93efab63e90b36c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.492730 14595 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:50:53.497365 14608 sys_catalog.cc:455] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "addf2a47fcc54adc93efab63e90b36c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.497535 14608 sys_catalog.cc:458] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:53.498781 14580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "addf2a47fcc54adc93efab63e90b36c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.498903 14580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:50:53.499315 14612 mvcc.cc:205] Tried to move back new op lower bound from 7333058369456492544 to 7333058369414742016. Current Snapshot: MvccSnapshot[applied={T|T < 7333058369456492544}]
I20260925 00:50:53.499702 14611 mvcc.cc:205] Tried to move back new op lower bound from 7333058369456492544 to 7333058369414742016. Current Snapshot: MvccSnapshot[applied={T|T < 7333058369456492544}]
I20260925 00:50:53.502864 14610 catalog_manager.cc:1347] Generated new cluster ID: d987de8a12804be5a5429ab7b4edb7fb
I20260925 00:50:53.502928 14610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:50:53.510008 14568 sys_catalog.cc:455] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "addf2a47fcc54adc93efab63e90b36c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.510129 14568 sys_catalog.cc:458] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:53.511235 14595 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "addf2a47fcc54adc93efab63e90b36c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1596273a1e4f27b3d496eb4a615a52" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44007 } } peers { permanent_uuid: "addf2a47fcc54adc93efab63e90b36c1" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42551 } } peers { permanent_uuid: "4497702ece0b4426abfafb3bd508f695" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 37273 } } }
I20260925 00:50:53.511353 14595 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:50:53.541632 14610 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:50:53.542991 14610 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:50:53.567284 14610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P addf2a47fcc54adc93efab63e90b36c1: Generated new TSK 0
I20260925 00:50:53.568212 14610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:50:54.382848 14593 catalog_manager.cc:1259] Loaded cluster ID: d987de8a12804be5a5429ab7b4edb7fb
I20260925 00:50:54.382933 14593 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695: loading cluster ID for follower catalog manager: success
I20260925 00:50:54.384632 14593 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695: acquiring CA information for follower catalog manager: success
I20260925 00:50:54.385847 14593 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4497702ece0b4426abfafb3bd508f695: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:50:54.394248 14606 catalog_manager.cc:1259] Loaded cluster ID: d987de8a12804be5a5429ab7b4edb7fb
I20260925 00:50:54.394347 14606 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52: loading cluster ID for follower catalog manager: success
I20260925 00:50:54.395947 14606 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52: acquiring CA information for follower catalog manager: success
I20260925 00:50:54.397053 14606 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf1596273a1e4f27b3d496eb4a615a52: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:50:54.637375 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 14335
I20260925 00:50:54.643592 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 14410
I20260925 00:50:54.649094 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 14485
2026-09-25T00:50:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3082 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:50:54 dist-test-slave-05m0 krb5kdc[14626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:50:54 dist-test-slave-05m0 krb5kdc[14626](info): set up 2 sockets
Sep 25 00:50:54 dist-test-slave-05m0 krb5kdc[14626](info): commencing operation
krb5kdc: starting...
W20260925 00:50:55.685768 997 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:50:55 dist-test-slave-05m0 krb5kdc[14626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:50:55.804277 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:34583
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:55.911336 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:55.911629 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:55.911707 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:55.915805 14639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:55.915894 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:55.915932 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:55.915992 14639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:55.916031 14639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:55.921149 14639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:34583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:55.922173 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:55.923148 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:55.931420 14647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:55.931529 14645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:55.931587 14639 server_base.cc:1034] running on GCE node
W20260925 00:50:55.931429 14644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:55.932281 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:55.933305 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:55.934474 14639 hybrid_clock.cc:648] HybridClock initialized: now 1790297455934460 us; error 25 us; skew 500 ppm
Sep 25 00:50:55 dist-test-slave-05m0 krb5kdc[14626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:55.937059 14639 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:50:55.937906 14639 webserver.cc:460] Webserver started at http://127.0.249.126:34197/ using document root <none> and password file <none>
I20260925 00:50:55.938392 14639 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:55.938449 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:55.938602 14639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:50:55.940518 14639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "29dd723c6e38437299227a3d506a6d88"
format_stamp: "Formatted at 2026-09-25 00:50:55 on dist-test-slave-05m0"
server_key: "29904f4ac60d1d176055dbd304a476bf"
server_key_iv: "a409e1d2ea8936444ff42d23a584495a"
server_key_version: "encryptionkey@0"
I20260925 00:50:55.941004 14639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "29dd723c6e38437299227a3d506a6d88"
format_stamp: "Formatted at 2026-09-25 00:50:55 on dist-test-slave-05m0"
server_key: "29904f4ac60d1d176055dbd304a476bf"
server_key_iv: "a409e1d2ea8936444ff42d23a584495a"
server_key_version: "encryptionkey@0"
I20260925 00:50:55.944466 14639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:50:55.946686 14654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:55.947851 14639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:50:55.947983 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "29dd723c6e38437299227a3d506a6d88"
format_stamp: "Formatted at 2026-09-25 00:50:55 on dist-test-slave-05m0"
server_key: "29904f4ac60d1d176055dbd304a476bf"
server_key_iv: "a409e1d2ea8936444ff42d23a584495a"
server_key_version: "encryptionkey@0"
I20260925 00:50:55.948089 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:55.967265 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:55.970646 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:55.970847 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:55.977548 14639 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:34583
I20260925 00:50:55.977581 14706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:34583 every 8 connection(s)
I20260925 00:50:55.978446 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:50:55.979954 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 14639
I20260925 00:50:55.980055 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:50:55.980352 997 external_mini_cluster.cc:1428] Setting key 03ba6560ec27373d4a7ff1f92e8e5c95
I20260925 00:50:55.981333 14707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:55.987849 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88: Bootstrap starting.
Sep 25 00:50:55 dist-test-slave-05m0 krb5kdc[14626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:55.990298 14707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:55.990993 14707 log.cc:824] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:55.993129 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88: No bootstrap required, opened a new log
I20260925 00:50:55.995045 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:55.981676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.0.249.126:34583)
0925 00:50:55.982069 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:55.982088 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:55.982130 (+ 42us) server_negotiation.cc:407] Connection header received
0925 00:50:55.982961 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:55.982984 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:55.983381 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:55.983726 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:55.984359 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:55.985845 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:55.986653 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:55.987327 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:55.987480 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:55.990133 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:55.990138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:55.990150 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:55.990174 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:55.992066 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:55.992467 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:55.992473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:55.992478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:55.992610 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:55.992858 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:55.992861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:55.992863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:55.993349 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:55.993617 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:55.993870 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 00:50:55.994150 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":118,"threads_started":1}
I20260925 00:50:55.997464 14707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd723c6e38437299227a3d506a6d88" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34583 } }
I20260925 00:50:55.997665 14707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:55.997740 14707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29dd723c6e38437299227a3d506a6d88, State: Initialized, Role: FOLLOWER
I20260925 00:50:55.997918 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:50:55.998214 14707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd723c6e38437299227a3d506a6d88" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34583 } }
I20260925 00:50:55.998337 14707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:50:55.998422 14707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:50:55.998540 14707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:55.999581 14707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd723c6e38437299227a3d506a6d88" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34583 } }
I20260925 00:50:55.999960 14707 leader_election.cc:302] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29dd723c6e38437299227a3d506a6d88; no voters:
I20260925 00:50:56.000247 14707 leader_election.cc:288] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:50:56.000469 14712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:50:56.000715 14712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [term 1 LEADER]: Becoming Leader. State: Replica: 29dd723c6e38437299227a3d506a6d88, State: Running, Role: LEADER
I20260925 00:50:56.001048 14712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd723c6e38437299227a3d506a6d88" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34583 } }
I20260925 00:50:56.001361 14707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:50:56.002755 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29dd723c6e38437299227a3d506a6d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd723c6e38437299227a3d506a6d88" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34583 } } }
I20260925 00:50:56.002797 14714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29dd723c6e38437299227a3d506a6d88. Latest consensus state: current_term: 1 leader_uuid: "29dd723c6e38437299227a3d506a6d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd723c6e38437299227a3d506a6d88" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34583 } } }
I20260925 00:50:56.002905 14714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:56.002905 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:56.003366 14724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:50:56.006171 14724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:50:56.011745 14724 catalog_manager.cc:1347] Generated new cluster ID: 88910c3603154705be1de4404c984ad3
I20260925 00:50:56.011811 14724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:50:56.022075 14724 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:50:56.023063 14724 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:50:56.028899 14724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29dd723c6e38437299227a3d506a6d88: Generated new TSK 0
I20260925 00:50:56.029695 14724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:50:56.053728 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 14639
2026-09-25T00:50:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1402 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:50:56 dist-test-slave-05m0 krb5kdc[14734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:50:56 dist-test-slave-05m0 krb5kdc[14734](info): set up 2 sockets
Sep 25 00:50:56 dist-test-slave-05m0 krb5kdc[14734](info): commencing operation
krb5kdc: starting...
W20260925 00:50:57.105718 997 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:50:57 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:50:57.224879 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36105
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:36105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:57.335181 14747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:57.335448 14747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:57.335487 14747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:57.339490 14747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:57.339551 14747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:57.339570 14747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:57.339587 14747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:57.339634 14747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:57.344974 14747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:36105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:57.346046 14747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:57.347010 14747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:57.355638 14752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:57.355659 14755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:57.355676 14747 server_base.cc:1034] running on GCE node
W20260925 00:50:57.355659 14753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:57.356525 14747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:57.357663 14747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:57.358846 14747 hybrid_clock.cc:648] HybridClock initialized: now 1790297457358815 us; error 41 us; skew 500 ppm
Sep 25 00:50:57 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:57.361541 14747 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:50:57.362416 14747 webserver.cc:460] Webserver started at http://127.0.249.126:40813/ using document root <none> and password file <none>
I20260925 00:50:57.362967 14747 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:57.363055 14747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:57.363284 14747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:50:57.365274 14747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "95da1db27d0e4209acf733e267888294"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "891a017b05ba7e28dec331e71487dc0b"
server_key_iv: "06934c921697bcfb3fdb310e0ef99392"
server_key_version: "encryptionkey@0"
I20260925 00:50:57.365895 14747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "95da1db27d0e4209acf733e267888294"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "891a017b05ba7e28dec331e71487dc0b"
server_key_iv: "06934c921697bcfb3fdb310e0ef99392"
server_key_version: "encryptionkey@0"
I20260925 00:50:57.369385 14747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:50:57.371922 14762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:57.373171 14747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:50:57.373311 14747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "95da1db27d0e4209acf733e267888294"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "891a017b05ba7e28dec331e71487dc0b"
server_key_iv: "06934c921697bcfb3fdb310e0ef99392"
server_key_version: "encryptionkey@0"
I20260925 00:50:57.373471 14747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:57.406412 14747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:57.409371 14747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:57.409631 14747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:57.416678 14747 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:36105
I20260925 00:50:57.416975 14814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:36105 every 8 connection(s)
I20260925 00:50:57.417728 14747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:50:57.420595 14815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:57.420979 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 14747
I20260925 00:50:57.421135 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:50:57.421486 997 external_mini_cluster.cc:1428] Setting key a3302b512f905402f4e91bcd3eadf621
I20260925 00:50:57.426697 14815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Bootstrap starting.
I20260925 00:50:57.429311 14815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:57.430253 14815 log.cc:824] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:50:57 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:57.432293 14815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: No bootstrap required, opened a new log
I20260925 00:50:57.435885 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:57.422995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.0.249.126:36105)
0925 00:50:57.423407 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:57.423422 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:57.423459 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:50:57.424119 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:57.424144 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:57.424486 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:57.424768 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:57.425671 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.426665 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.427803 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.428290 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.428414 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:57.431349 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:57.431355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:57.431367 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:57.431392 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:57.433087 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:57.433615 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:57.433620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:57.433624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:57.433733 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:57.434023 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:57.434026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:57.434028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:57.434410 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:57.434607 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:57.434906 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 00:50:57.435182 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":135,"threads_started":1}
I20260925 00:50:57.435756 14815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:57.436028 14815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:57.436093 14815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95da1db27d0e4209acf733e267888294, State: Initialized, Role: FOLLOWER
I20260925 00:50:57.436496 14815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:57.436597 14815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:50:57.436633 14815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:50:57.436692 14815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:57.437625 14815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:57.437955 14815 leader_election.cc:302] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95da1db27d0e4209acf733e267888294; no voters:
I20260925 00:50:57.438283 14815 leader_election.cc:288] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:50:57.438401 14820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:50:57.438644 14820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 1 LEADER]: Becoming Leader. State: Replica: 95da1db27d0e4209acf733e267888294, State: Running, Role: LEADER
I20260925 00:50:57.438939 14820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:57.439491 14815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:50:57.440487 14822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95da1db27d0e4209acf733e267888294. Latest consensus state: current_term: 1 leader_uuid: "95da1db27d0e4209acf733e267888294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } } }
I20260925 00:50:57.440611 14822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:57.440932 14821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95da1db27d0e4209acf733e267888294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } } }
I20260925 00:50:57.441020 14821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:57.441104 14829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:50:57.443892 14829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:50:57.449775 14829 catalog_manager.cc:1347] Generated new cluster ID: 7ff1d788dadb4e7f9e5b18ec26aecc5a
I20260925 00:50:57.449864 14829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:50:57.471185 14829 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:50:57.472487 14829 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:50:57.485255 14829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Generated new TSK 0
I20260925 00:50:57.486024 14829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:50:57.525007 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36105
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:57.635691 14841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:57.635933 14841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:57.635974 14841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:57.640004 14841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:57.640059 14841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:57.640116 14841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:50:57.645480 14841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:57.646394 14841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:57.647339 14841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:57.656940 14846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:57.656960 14847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:57.656948 14849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:57.657208 14841 server_base.cc:1034] running on GCE node
I20260925 00:50:57.657735 14841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:57.658411 14841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:57.659582 14841 hybrid_clock.cc:648] HybridClock initialized: now 1790297457659561 us; error 36 us; skew 500 ppm
Sep 25 00:50:57 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:57.662302 14841 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:50:57.663189 14841 webserver.cc:460] Webserver started at http://127.0.249.65:37631/ using document root <none> and password file <none>
I20260925 00:50:57.663744 14841 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:57.663837 14841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:57.664072 14841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:50:57.666100 14841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "604494106bcd4573aabbfb1b93b656e2"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "87ed42c0b031b86cab56017a5bc7c06c"
server_key_iv: "56f7c97cc5f8790c43b8904c7992a50f"
server_key_version: "encryptionkey@0"
I20260925 00:50:57.666661 14841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "604494106bcd4573aabbfb1b93b656e2"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "87ed42c0b031b86cab56017a5bc7c06c"
server_key_iv: "56f7c97cc5f8790c43b8904c7992a50f"
server_key_version: "encryptionkey@0"
I20260925 00:50:57.670226 14841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:50:57.672640 14856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:57.673866 14841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 00:50:57.673998 14841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "604494106bcd4573aabbfb1b93b656e2"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "87ed42c0b031b86cab56017a5bc7c06c"
server_key_iv: "56f7c97cc5f8790c43b8904c7992a50f"
server_key_version: "encryptionkey@0"
I20260925 00:50:57.674118 14841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:57.685961 14841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:57.688740 14841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:57.688939 14841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:57.689545 14841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:57.690356 14841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:57.690429 14841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:57.690498 14841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:57.690552 14841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:57.699225 14841 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:35061
I20260925 00:50:57.699260 14969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:35061 every 8 connection(s)
I20260925 00:50:57.700065 14841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:50:57.700886 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 14841
I20260925 00:50:57.701013 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:50:57.701309 997 external_mini_cluster.cc:1428] Setting key adc768ea9a1b9246817c2b5071edea46
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
Sep 25 00:50:57 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
I20260925 00:50:57.715941 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:57.701951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:42265 (local address 127.0.249.126:36105)
0925 00:50:57.702097 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:57.702101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:57.703175 (+ 1074us) server_negotiation.cc:407] Connection header received
0925 00:50:57.704965 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:57.704969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:57.705023 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:57.705102 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:57.706634 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.707289 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.709124 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.709355 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.709511 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:57.712280 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:57.712315 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:57.712317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:57.712365 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:57.713928 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:57.714440 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:57.714445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:57.714447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:57.714528 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:57.714877 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:57.714880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:57.714882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:57.715060 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:57.715163 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:57.715634 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 00:50:57.715758 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:50:57.716696 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:57.702400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:42265)
0925 00:50:57.703007 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:57.703065 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:57.704765 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:57.705262 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:57.705269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:57.705792 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:57.706463 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:57.706485 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.707434 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.707440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:57.708985 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:57.709000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.709568 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.709574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:57.709791 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:57.710435 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:57.710455 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:57.712114 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:57.714048 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:57.714055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:57.714069 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:57.714331 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:57.714628 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:57.714631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:57.714633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:57.714769 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:57.715162 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:57.715170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:57.715408 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 00:50:57.715595 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 00:50:57.715906 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":124,"threads_started":1}
I20260925 00:50:57.717981 14970 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36105
I20260925 00:50:57.718247 14970 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:57.718869 14970 heartbeater.cc:511] Master 127.0.249.126:36105 requested a full tablet report, sending...
I20260925 00:50:57.720623 14779 ts_manager.cc:194] Registered new tserver with Master: 604494106bcd4573aabbfb1b93b656e2 (127.0.249.65:35061)
I20260925 00:50:57.722179 14779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:42265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:50:57.728683 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36105
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:57.838491 14975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:57.838750 14975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:57.838789 14975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:57.843003 14975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:57.843061 14975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:57.843117 14975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:50:57.848088 14975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:57.848987 14975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:57.849923 14975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:57.859364 14981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:57.859390 14980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:57.859592 14975 server_base.cc:1034] running on GCE node
W20260925 00:50:57.859390 14983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:57.860189 14975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:57.860821 14975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:57.862001 14975 hybrid_clock.cc:648] HybridClock initialized: now 1790297457861974 us; error 38 us; skew 500 ppm
Sep 25 00:50:57 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:57.864826 14975 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:50:57.865741 14975 webserver.cc:460] Webserver started at http://127.0.249.66:32779/ using document root <none> and password file <none>
I20260925 00:50:57.866290 14975 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:57.866379 14975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:57.866598 14975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:50:57.868505 14975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "8f80559ea0ce4b1ba9f4a996726e7392"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "9c344ff38fa5e80fec72cce042d7af5f"
server_key_iv: "bcc7172639a7ec8a87779ba188283ce4"
server_key_version: "encryptionkey@0"
I20260925 00:50:57.869041 14975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "8f80559ea0ce4b1ba9f4a996726e7392"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "9c344ff38fa5e80fec72cce042d7af5f"
server_key_iv: "bcc7172639a7ec8a87779ba188283ce4"
server_key_version: "encryptionkey@0"
I20260925 00:50:57.872675 14975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 00:50:57.875043 14990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:57.876255 14975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:50:57.876401 14975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "8f80559ea0ce4b1ba9f4a996726e7392"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "9c344ff38fa5e80fec72cce042d7af5f"
server_key_iv: "bcc7172639a7ec8a87779ba188283ce4"
server_key_version: "encryptionkey@0"
I20260925 00:50:57.876513 14975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:57.892225 14975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:57.895098 14975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:57.895299 14975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:57.895880 14975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:57.896684 14975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:57.896754 14975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:57.896824 14975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:57.896879 14975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:57.905858 14975 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:40617
I20260925 00:50:57.905884 15103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:40617 every 8 connection(s)
I20260925 00:50:57.906711 14975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:50:57.914789 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 14975
I20260925 00:50:57.914968 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:50:57.915248 997 external_mini_cluster.cc:1428] Setting key b61e65d9a58fc225c658e6ca68fd8575
Sep 25 00:50:57 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:57.919880 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:57.908458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:40597 (local address 127.0.249.126:36105)
0925 00:50:57.908591 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:57.908595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:57.909435 (+ 840us) server_negotiation.cc:407] Connection header received
0925 00:50:57.910692 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:57.910696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:57.910753 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:57.910870 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:57.911999 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.912525 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.913658 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.913887 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.913999 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:57.916484 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:57.916493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:57.916495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:57.916517 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:57.917898 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:57.918425 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:57.918430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:57.918432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:57.918512 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:57.918837 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:57.918841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:57.918845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:57.919017 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:57.919123 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:57.919663 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 00:50:57.919758 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:50:57.921020 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:57.908837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:40597)
0925 00:50:57.909260 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:57.909302 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:57.910471 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:57.911045 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:57.911051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:57.911405 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:57.911852 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:57.911864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.912646 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.912650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:57.913530 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:57.913538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:57.914033 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:57.914037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:57.914204 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:57.914811 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:57.914826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:57.916326 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:57.918008 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:57.918013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:57.918026 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:57.918320 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:57.918602 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:57.918606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:57.918608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:57.918742 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:57.919121 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:57.919127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:57.919405 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 00:50:57.919604 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 00:50:57.919946 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":103,"threads_started":1}
I20260925 00:50:57.922257 15104 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36105
I20260925 00:50:57.922529 15104 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:57.923090 15104 heartbeater.cc:511] Master 127.0.249.126:36105 requested a full tablet report, sending...
I20260925 00:50:57.924161 14779 ts_manager.cc:194] Registered new tserver with Master: 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617)
I20260925 00:50:57.924733 14779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:40597
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:50:57.942355 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36105
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:58.051514 15109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:58.051798 15109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:58.051875 15109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:58.056097 15109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:58.056174 15109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:58.056265 15109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:50:58.061540 15109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:58.062532 15109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:58.063512 15109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:58.072919 15117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:58.072961 15114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:58.073136 15109 server_base.cc:1034] running on GCE node
W20260925 00:50:58.072961 15115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:58.073798 15109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:58.074386 15109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:58.075542 15109 hybrid_clock.cc:648] HybridClock initialized: now 1790297458075529 us; error 30 us; skew 500 ppm
Sep 25 00:50:58 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:58.078305 15109 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:50:58.079187 15109 webserver.cc:460] Webserver started at http://127.0.249.67:42835/ using document root <none> and password file <none>
I20260925 00:50:58.079767 15109 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:58.079859 15109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:58.080065 15109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:50:58.082079 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "9fda2bc1e81248d7ba08065a7190b50f"
format_stamp: "Formatted at 2026-09-25 00:50:58 on dist-test-slave-05m0"
server_key: "077dc667c09369cab021b7f800fc4383"
server_key_iv: "c28b6c2c1a86602c318d542eca513401"
server_key_version: "encryptionkey@0"
I20260925 00:50:58.082615 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "9fda2bc1e81248d7ba08065a7190b50f"
format_stamp: "Formatted at 2026-09-25 00:50:58 on dist-test-slave-05m0"
server_key: "077dc667c09369cab021b7f800fc4383"
server_key_iv: "c28b6c2c1a86602c318d542eca513401"
server_key_version: "encryptionkey@0"
I20260925 00:50:58.086074 15109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:50:58.088282 15124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:58.089459 15109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:50:58.089578 15109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "9fda2bc1e81248d7ba08065a7190b50f"
format_stamp: "Formatted at 2026-09-25 00:50:58 on dist-test-slave-05m0"
server_key: "077dc667c09369cab021b7f800fc4383"
server_key_iv: "c28b6c2c1a86602c318d542eca513401"
server_key_version: "encryptionkey@0"
I20260925 00:50:58.089681 15109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:58.112586 15109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:58.115415 15109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:58.115620 15109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:58.116212 15109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:58.117012 15109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:58.117079 15109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:58.117151 15109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:58.117187 15109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:58.125855 15109 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:42289
I20260925 00:50:58.125885 15237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:42289 every 8 connection(s)
I20260925 00:50:58.126688 15109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:50:58.128535 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 15109
I20260925 00:50:58.128643 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:50:58.128901 997 external_mini_cluster.cc:1428] Setting key 2d57ec4deab943e09a0b9dd22ad669a9
Sep 25 00:50:58 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:58.139950 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.128468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:48813 (local address 127.0.249.126:36105)
0925 00:50:58.128637 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.128642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.129669 (+ 1027us) server_negotiation.cc:407] Connection header received
0925 00:50:58.131045 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.131048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.131089 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.131153 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:58.132315 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.132828 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.134010 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.134216 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.134289 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.136892 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:58.136895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:58.136897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:58.136919 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:58.138227 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:58.138669 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:58.138672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:58.138673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:58.138739 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:58.139036 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:58.139038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:58.139039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:58.139191 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:58.139260 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.139672 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.139762 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 00:50:58.140793 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.128965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:48813)
0925 00:50:58.129504 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:58.129562 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:58.130867 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:58.131329 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:58.131336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:58.131698 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:58.132195 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.132209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.132929 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.132932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.133906 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.133916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.134344 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.134347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.134486 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.135169 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:58.135186 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:58.136720 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:58.138327 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:58.138331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:58.138343 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:58.138586 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:58.138818 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:58.138820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:58.138821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:58.138960 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:58.139275 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:58.139280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:58.139481 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 00:50:58.139627 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 00:50:58.139989 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
I20260925 00:50:58.142016 15238 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36105
I20260925 00:50:58.142279 15238 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:58.142792 15238 heartbeater.cc:511] Master 127.0.249.126:36105 requested a full tablet report, sending...
I20260925 00:50:58.143692 14778 ts_manager.cc:194] Registered new tserver with Master: 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289)
I20260925 00:50:58.144217 14778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:48813
I20260925 00:50:58.154412 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:50:58.162957 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.156262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.249.126:36105)
0925 00:50:58.156391 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.156395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.156580 (+ 185us) server_negotiation.cc:407] Connection header received
0925 00:50:58.156672 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.156674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.156710 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.156775 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:58.157448 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.157995 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.158769 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.158988 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.159059 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.160262 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:58.160265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:58.160267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:58.160288 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:58.161552 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:58.161938 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:58.161942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:58.161943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:58.162033 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:58.162264 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:58.162266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:58.162267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:58.162461 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:58.162562 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.162749 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.162833 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:50:58.165725 14778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:50:58.168561 14778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:50:58.182876 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.176806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:40617 (local address 127.0.0.1:57654)
0925 00:50:58.177161 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:58.177216 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:58.177412 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:58.178323 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:58.178331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:58.178362 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:58.178470 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.178475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.179744 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.179747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.181174 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.181183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.182325 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.182332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.182448 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.182486 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 00:50:58.182598 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 00:50:58.182714 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":4,"thread_start_us":78,"threads_started":1}
I20260925 00:50:58.182909 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.176921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.0.249.66:40617)
0925 00:50:58.177406 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.177420 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.177447 (+ 27us) server_negotiation.cc:407] Connection header received
0925 00:50:58.177685 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.177693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.177936 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.178161 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:58.178584 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.179603 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.181294 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.182191 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.182300 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.182437 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.182640 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.182748 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":122,"threads_started":1}
I20260925 00:50:58.183207 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.177710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.0.249.65:35061)
0925 00:50:58.178456 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.178467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.178485 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:50:58.178544 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.178548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.178675 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.178822 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:58.179106 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.180083 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.181687 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.182273 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.182379 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.182519 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.182951 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.183057 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":81,"threads_started":1}
I20260925 00:50:58.183898 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.177252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:35061 (local address 127.0.0.1:52778)
0925 00:50:58.178099 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:58.178115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:58.178200 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:58.178800 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:58.178803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:58.178828 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:58.178949 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.178957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.180335 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.180339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.181565 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.181575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.182584 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.182590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.182686 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.182706 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:50:58.182768 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 00:50:58.182822 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":103,"threads_started":1}
I20260925 00:50:58.184594 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.177543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42289 (local address 127.0.0.1:57140)
0925 00:50:58.179176 (+ 1633us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:58.179187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:58.179287 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:58.179898 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:58.179901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:58.179917 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:58.180060 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.180066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.181529 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.181532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.182629 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.182639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.184154 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.184157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.184264 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.184286 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:50:58.184403 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 00:50:58.184474 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1450,"mutex_wait_us":37,"thread_start_us":85,"threads_started":1}
I20260925 00:50:58.184650 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.177953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.0.249.67:42289)
0925 00:50:58.178402 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.178413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.179270 (+ 857us) server_negotiation.cc:407] Connection header received
0925 00:50:58.179433 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.179437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.179603 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.179744 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:58.180194 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.181420 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.182764 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.183853 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.183949 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.184092 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.184433 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.184517 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":148,"threads_started":1}
I20260925 00:50:58.186123 15038 tablet_service.cc:1467] Processing CreateTablet for tablet a6c57a3545704b70827ab8365f047e87 (DEFAULT_TABLE table=test-table [id=89010e5037e945e2989ccdec9834ee0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:58.186643 14904 tablet_service.cc:1467] Processing CreateTablet for tablet a6c57a3545704b70827ab8365f047e87 (DEFAULT_TABLE table=test-table [id=89010e5037e945e2989ccdec9834ee0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:58.187173 15038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c57a3545704b70827ab8365f047e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:58.187613 14904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c57a3545704b70827ab8365f047e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:58.188428 15172 tablet_service.cc:1467] Processing CreateTablet for tablet a6c57a3545704b70827ab8365f047e87 (DEFAULT_TABLE table=test-table [id=89010e5037e945e2989ccdec9834ee0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:58.189358 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c57a3545704b70827ab8365f047e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:58.193524 15255 tablet_bootstrap.cc:492] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Bootstrap starting.
I20260925 00:50:58.194666 15256 tablet_bootstrap.cc:492] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: Bootstrap starting.
I20260925 00:50:58.195951 15255 tablet_bootstrap.cc:654] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:58.195988 15257 tablet_bootstrap.cc:492] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: Bootstrap starting.
I20260925 00:50:58.197059 15256 tablet_bootstrap.cc:654] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:58.197216 15255 log.cc:824] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:58.197934 15256 log.cc:824] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:58.197990 15257 tablet_bootstrap.cc:654] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:58.199117 15257 log.cc:824] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:58.199616 15255 tablet_bootstrap.cc:492] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: No bootstrap required, opened a new log
I20260925 00:50:58.199615 15256 tablet_bootstrap.cc:492] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: No bootstrap required, opened a new log
I20260925 00:50:58.199826 15256 ts_tablet_manager.cc:1397] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:50:58.199824 15255 ts_tablet_manager.cc:1397] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:50:58.200927 15257 tablet_bootstrap.cc:492] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: No bootstrap required, opened a new log
I20260925 00:50:58.201102 15257 ts_tablet_manager.cc:1397] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:50:58.202737 15256 raft_consensus.cc:348] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.202971 15256 raft_consensus.cc:374] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:58.203030 15256 raft_consensus.cc:729] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f80559ea0ce4b1ba9f4a996726e7392, State: Initialized, Role: FOLLOWER
I20260925 00:50:58.203464 15256 consensus_queue.cc:260] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.203676 15255 raft_consensus.cc:348] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.203917 15255 raft_consensus.cc:374] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:58.203972 15255 raft_consensus.cc:729] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604494106bcd4573aabbfb1b93b656e2, State: Initialized, Role: FOLLOWER
I20260925 00:50:58.204262 15104 heartbeater.cc:503] Master 127.0.249.126:36105 was elected leader, sending a full tablet report...
I20260925 00:50:58.204497 15256 ts_tablet_manager.cc:1428] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:50:58.204779 15257 raft_consensus.cc:348] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.205016 15257 raft_consensus.cc:374] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:58.205063 15257 raft_consensus.cc:729] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fda2bc1e81248d7ba08065a7190b50f, State: Initialized, Role: FOLLOWER
I20260925 00:50:58.205111 15255 consensus_queue.cc:260] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.205503 15257 consensus_queue.cc:260] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.205862 14970 heartbeater.cc:503] Master 127.0.249.126:36105 was elected leader, sending a full tablet report...
I20260925 00:50:58.206080 15255 ts_tablet_manager.cc:1428] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260925 00:50:58.206349 15257 ts_tablet_manager.cc:1428] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:50:58.206595 15238 heartbeater.cc:503] Master 127.0.249.126:36105 was elected leader, sending a full tablet report...
I20260925 00:50:58.320109 15261 raft_consensus.cc:482] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:50:58.320312 15261 raft_consensus.cc:504] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.321605 15261 leader_election.cc:288] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289), 604494106bcd4573aabbfb1b93b656e2 (127.0.249.65:35061)
I20260925 00:50:58.326150 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.321932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:35061 (local address 127.0.249.66:52625)
0925 00:50:58.322317 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:58.322329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:58.322454 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:58.322722 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:58.322729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:58.322743 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:58.322826 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.322830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.323722 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.323724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.324972 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.324984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.325691 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.325695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.325804 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.325836 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 00:50:58.325910 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 00:50:58.325996 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":57,"thread_start_us":175,"threads_started":1}
I20260925 00:50:58.326164 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.322020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:52625 (local address 127.0.249.65:35061)
0925 00:50:58.322154 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.322160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.322441 (+ 281us) server_negotiation.cc:407] Connection header received
0925 00:50:58.322562 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.322564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.322604 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.322695 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:58.322934 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.323631 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.325100 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.325572 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.325667 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.325705 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.325946 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.326027 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:50:58.326292 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.321788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42289 (local address 127.0.249.66:38467)
0925 00:50:58.321930 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:58.321946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:58.322046 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:58.322439 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:58.322442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:58.322456 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:58.322591 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.322596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.323588 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.323590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.324977 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.324984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.325859 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.325862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.326000 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.326021 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:50:58.326098 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 00:50:58.326166 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":52}
I20260925 00:50:58.326363 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.322081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:38467 (local address 127.0.249.67:42289)
0925 00:50:58.322221 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.322224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.322236 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:50:58.322282 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.322284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.322320 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.322385 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:58.322704 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.323479 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.325142 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.325741 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.325873 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.325931 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.326132 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.326219 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:50:58.327006 14924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c57a3545704b70827ab8365f047e87" candidate_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604494106bcd4573aabbfb1b93b656e2" is_pre_election: true
I20260925 00:50:58.327191 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c57a3545704b70827ab8365f047e87" candidate_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fda2bc1e81248d7ba08065a7190b50f" is_pre_election: true
I20260925 00:50:58.327343 14924 raft_consensus.cc:2454] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f80559ea0ce4b1ba9f4a996726e7392 in term 0.
I20260925 00:50:58.327512 15192 raft_consensus.cc:2454] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f80559ea0ce4b1ba9f4a996726e7392 in term 0.
I20260925 00:50:58.327888 14991 leader_election.cc:302] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604494106bcd4573aabbfb1b93b656e2, 8f80559ea0ce4b1ba9f4a996726e7392; no voters:
I20260925 00:50:58.328212 15261 raft_consensus.cc:2781] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:50:58.328297 15261 raft_consensus.cc:482] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:50:58.328341 15261 raft_consensus.cc:3037] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:58.329371 15261 raft_consensus.cc:504] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.329753 15261 leader_election.cc:288] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [CANDIDATE]: Term 1 election: Requested vote from peers 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289), 604494106bcd4573aabbfb1b93b656e2 (127.0.249.65:35061)
I20260925 00:50:58.330121 14924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c57a3545704b70827ab8365f047e87" candidate_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604494106bcd4573aabbfb1b93b656e2"
I20260925 00:50:58.330233 14924 raft_consensus.cc:3037] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:58.331230 14924 raft_consensus.cc:2454] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f80559ea0ce4b1ba9f4a996726e7392 in term 1.
I20260925 00:50:58.331518 14991 leader_election.cc:302] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604494106bcd4573aabbfb1b93b656e2, 8f80559ea0ce4b1ba9f4a996726e7392; no voters:
I20260925 00:50:58.331492 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c57a3545704b70827ab8365f047e87" candidate_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fda2bc1e81248d7ba08065a7190b50f"
I20260925 00:50:58.331657 15192 raft_consensus.cc:3037] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:58.331702 15261 raft_consensus.cc:2781] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:50:58.331956 15261 raft_consensus.cc:686] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 LEADER]: Becoming Leader. State: Replica: 8f80559ea0ce4b1ba9f4a996726e7392, State: Running, Role: LEADER
I20260925 00:50:58.332311 15261 consensus_queue.cc:237] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.332798 15192 raft_consensus.cc:2454] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f80559ea0ce4b1ba9f4a996726e7392 in term 1.
I20260925 00:50:58.335536 14778 catalog_manager.cc:5495] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66). New cstate: current_term: 1 leader_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } health_report { overall_health: UNKNOWN } } }
I20260925 00:50:58.369813 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.366291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.0.249.66:40617)
0925 00:50:58.366435 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.366440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.366455 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:50:58.366569 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.366572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.366627 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.366743 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:50:58.366991 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.367484 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.368338 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.368555 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.368665 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.368963 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:50:58.369362 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:50:58.369539 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.369584 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.369640 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:50:58.375861 15192 raft_consensus.cc:1261] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [term 1 FOLLOWER]: Refusing update from remote peer 8f80559ea0ce4b1ba9f4a996726e7392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:58.375865 14924 raft_consensus.cc:1261] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Refusing update from remote peer 8f80559ea0ce4b1ba9f4a996726e7392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:58.376652 15265 consensus_queue.cc:1035] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:50:58.376888 15261 consensus_queue.cc:1035] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 00:50:58.377583 15239 tablet.cc:2369] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:50:58.385619 15269 mvcc.cc:205] Tried to move back new op lower bound from 7333058389499936768 to 7333058389332135936. Current Snapshot: MvccSnapshot[applied={T|T < 7333058389499936768}]
I20260925 00:50:58.388223 15270 mvcc.cc:205] Tried to move back new op lower bound from 7333058389499936768 to 7333058389332135936. Current Snapshot: MvccSnapshot[applied={T|T < 7333058389499936768}]
I20260925 00:50:58.394178 14779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:50:58.394330 14779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:50:58.396979 14779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Sending DeleteTablet for 3 replicas of tablet a6c57a3545704b70827ab8365f047e87
I20260925 00:50:58.397912 15038 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c57a3545704b70827ab8365f047e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:58 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:57654
I20260925 00:50:58.398069 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 14841
I20260925 00:50:58.398069 15172 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c57a3545704b70827ab8365f047e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:58 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:57140
I20260925 00:50:58.399096 15278 tablet_replica.cc:330] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: stopping tablet replica
I20260925 00:50:58.399416 15278 raft_consensus.cc:2229] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:50:58.399768 15278 raft_consensus.cc:2258] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:50:58.401320 15278 ts_tablet_manager.cc:1905] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:50:58.404834 15278 ts_tablet_manager.cc:1918] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:50:58.404949 15278 log.cc:1196] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/a6c57a3545704b70827ab8365f047e87
I20260925 00:50:58.405244 15279 tablet_replica.cc:330] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: stopping tablet replica
I20260925 00:50:58.405261 15278 ts_tablet_manager.cc:1939] T a6c57a3545704b70827ab8365f047e87 P 8f80559ea0ce4b1ba9f4a996726e7392: Deleting consensus metadata
I20260925 00:50:58.405656 15279 raft_consensus.cc:2229] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:50:58.406001 15279 raft_consensus.cc:2258] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:50:58.406843 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 14975
I20260925 00:50:58.406829 14766 catalog_manager.cc:4841] TS 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617): tablet a6c57a3545704b70827ab8365f047e87 (table test-table [id=89010e5037e945e2989ccdec9834ee0d]) successfully deleted
W20260925 00:50:58.407083 14763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:35061 (error 108)
I20260925 00:50:58.409695 15279 ts_tablet_manager.cc:1905] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:50:58.410169 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.409911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:35061 (local address 127.0.0.1:52788)
0925 00:50:58.410032 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:58.410101 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 00:50:58.410378 14763 catalog_manager.cc:4584] TS 604494106bcd4573aabbfb1b93b656e2 (127.0.249.65:35061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6c57a3545704b70827ab8365f047e87: Network error: Client connection negotiation failed: client connection to 127.0.249.65:35061: connect: Connection refused (error 111)
I20260925 00:50:58.412276 15279 ts_tablet_manager.cc:1918] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:50:58.412374 15279 log.cc:1196] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/a6c57a3545704b70827ab8365f047e87
I20260925 00:50:58.412693 15279 ts_tablet_manager.cc:1939] T a6c57a3545704b70827ab8365f047e87 P 9fda2bc1e81248d7ba08065a7190b50f: Deleting consensus metadata
I20260925 00:50:58.413678 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 15109
I20260925 00:50:58.419186 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 14747
I20260925 00:50:58.425649 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36105
--webserver_interface=127.0.249.126
--webserver_port=40813
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:36105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:58.534811 15281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:58.535082 15281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:58.535122 15281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:58.539192 15281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:58.539250 15281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:58.539270 15281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:58.539287 15281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:58.539337 15281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:58.544636 15281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:36105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:58.545771 15281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:58.546801 15281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:58.559896 15287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:58.560073 15281 server_base.cc:1034] running on GCE node
W20260925 00:50:58.559937 15290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:58.559929 15288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:58.560992 15281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:58.562263 15281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:58.563426 15281 hybrid_clock.cc:648] HybridClock initialized: now 1790297458563410 us; error 27 us; skew 500 ppm
Sep 25 00:50:58 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:58.570086 15281 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:50:58.571080 15281 webserver.cc:460] Webserver started at http://127.0.249.126:40813/ using document root <none> and password file <none>
I20260925 00:50:58.571637 15281 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:58.571710 15281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:58.575775 15281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 00:50:58.577930 15297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:58.579164 15281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:50:58.579311 15281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "95da1db27d0e4209acf733e267888294"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "891a017b05ba7e28dec331e71487dc0b"
server_key_iv: "06934c921697bcfb3fdb310e0ef99392"
server_key_version: "encryptionkey@0"
I20260925 00:50:58.579916 15281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:58.607872 15281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:58.610903 15281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:58.611114 15281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:58.618335 15281 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:36105
I20260925 00:50:58.618613 15349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:36105 every 8 connection(s)
I20260925 00:50:58.619424 15281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:50:58.621783 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 15281
I20260925 00:50:58.622238 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:35061
--local_ip_for_outbound_sockets=127.0.249.65
--tserver_master_addrs=127.0.249.126:36105
--webserver_port=37631
--webserver_interface=127.0.249.65
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:50:58.624326 15350 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:50:58.626116 15350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Bootstrap starting.
I20260925 00:50:58.642916 15350 log.cc:824] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:58.651475 15350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:50:58.651855 15350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Bootstrap complete.
I20260925 00:50:58.656536 15350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:58.656931 15350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95da1db27d0e4209acf733e267888294, State: Initialized, Role: FOLLOWER
I20260925 00:50:58.657469 15350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:58.657582 15350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:50:58.657666 15350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:50:58.657759 15350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:50:58.659631 15350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:58.659968 15350 leader_election.cc:302] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95da1db27d0e4209acf733e267888294; no voters:
I20260925 00:50:58.660272 15350 leader_election.cc:288] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 00:50:58.660413 15354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:50:58.660648 15354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 2 LEADER]: Becoming Leader. State: Replica: 95da1db27d0e4209acf733e267888294, State: Running, Role: LEADER
I20260925 00:50:58.660951 15354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:58.661557 15350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:50:58.661948 15354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95da1db27d0e4209acf733e267888294. Latest consensus state: current_term: 2 leader_uuid: "95da1db27d0e4209acf733e267888294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } } }
I20260925 00:50:58.662056 15354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:58.662457 15355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "95da1db27d0e4209acf733e267888294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } } }
I20260925 00:50:58.662647 15355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:58.663865 15358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:50:58.667761 15358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:50:58.668264 15358 catalog_manager.cc:1259] Loaded cluster ID: 7ff1d788dadb4e7f9e5b18ec26aecc5a
I20260925 00:50:58.668339 15358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:50:58.669443 15358 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 00:50:58.672443 15358 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 00:50:58.678578 15358 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:50:58.679257 15358 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Loaded TSK: 0
I20260925 00:50:58.680199 15358 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:50:58.743777 15352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:58.744079 15352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:58.744158 15352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:58.748335 15352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:58.748428 15352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:58.748522 15352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:50:58.753924 15352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:35061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=37631
--tserver_master_addrs=127.0.249.126:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:58.755115 15352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:58.756319 15352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:58.766271 15377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:58.766217 15378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:58.766232 15380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:58.766633 15352 server_base.cc:1034] running on GCE node
I20260925 00:50:58.767017 15352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:58.767795 15352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:58.768982 15352 hybrid_clock.cc:648] HybridClock initialized: now 1790297458768964 us; error 37 us; skew 500 ppm
Sep 25 00:50:58 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:58.771831 15352 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:50:58.772760 15352 webserver.cc:460] Webserver started at http://127.0.249.65:37631/ using document root <none> and password file <none>
I20260925 00:50:58.773377 15352 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:58.773526 15352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:58.777444 15352 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 00:50:58.779726 15387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:58.780995 15352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:50:58.781127 15352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "604494106bcd4573aabbfb1b93b656e2"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "87ed42c0b031b86cab56017a5bc7c06c"
server_key_iv: "56f7c97cc5f8790c43b8904c7992a50f"
server_key_version: "encryptionkey@0"
I20260925 00:50:58.781721 15352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:58.812197 15352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:58.815629 15352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:58.815840 15352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:58.816449 15352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:58.817646 15394 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:50:58.821650 15352 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:50:58.821720 15352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260925 00:50:58.821800 15352 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:50:58.823463 15352 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:50:58.823532 15352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 00:50:58.823784 15394 tablet_bootstrap.cc:492] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Bootstrap starting.
I20260925 00:50:58.834259 15352 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:35061
I20260925 00:50:58.834358 15501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:35061 every 8 connection(s)
I20260925 00:50:58.835203 15352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:50:58.839107 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 15352
I20260925 00:50:58.839588 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:40617
--local_ip_for_outbound_sockets=127.0.249.66
--tserver_master_addrs=127.0.249.126:36105
--webserver_port=32779
--webserver_interface=127.0.249.66
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:50:58.842131 15394 log.cc:824] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:50:58 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:58.851562 15394 tablet_bootstrap.cc:492] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:50:58.851979 15394 tablet_bootstrap.cc:492] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Bootstrap complete.
I20260925 00:50:58.852423 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.837232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:49267 (local address 127.0.249.126:36105)
0925 00:50:58.837637 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.837656 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.838001 (+ 345us) server_negotiation.cc:407] Connection header received
0925 00:50:58.839563 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.839579 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.839997 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.840374 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:58.841740 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.843213 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.844474 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.844943 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.845129 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.847544 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:58.847549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:58.847560 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:58.847587 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:58.849286 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:58.849850 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:58.849855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:58.849859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:58.849970 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:58.850323 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:58.850333 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:58.850334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:58.850650 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:58.850797 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.851309 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.851574 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":153,"threads_started":1}
I20260925 00:50:58.852439 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.837183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:49267)
0925 00:50:58.837818 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:58.837873 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:58.839077 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:58.840464 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:58.840472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:58.840921 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:58.841550 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.841571 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.843386 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.843389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.844327 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.844337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.845089 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.845095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.845282 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.845930 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:58.845946 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:58.847355 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:58.849443 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:58.849448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:58.849461 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:58.849714 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:58.850076 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:58.850079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:58.850081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:58.850219 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:58.850775 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:58.850780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:58.851012 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 00:50:58.851185 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 00:50:58.851512 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":168,"threads_started":1}
I20260925 00:50:58.853427 15394 ts_tablet_manager.cc:1397] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.008s
I20260925 00:50:58.855415 15502 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36105
I20260925 00:50:58.855677 15502 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:58.856210 15502 heartbeater.cc:511] Master 127.0.249.126:36105 requested a full tablet report, sending...
I20260925 00:50:58.856424 15394 raft_consensus.cc:348] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.856921 15394 raft_consensus.cc:729] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604494106bcd4573aabbfb1b93b656e2, State: Initialized, Role: FOLLOWER
I20260925 00:50:58.857411 15394 consensus_queue.cc:260] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:58.858381 15394 ts_tablet_manager.cc:1428] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:50:58.860376 15314 ts_manager.cc:194] Registered new tserver with Master: 604494106bcd4573aabbfb1b93b656e2 (127.0.249.65:35061)
I20260925 00:50:58.863948 15314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:49267
I20260925 00:50:58.866427 15502 heartbeater.cc:503] Master 127.0.249.126:36105 was elected leader, sending a full tablet report...
Sep 25 00:50:58 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:50:58.873916 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.863274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:35061 (local address 127.0.0.1:52792)
0925 00:50:58.863875 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:58.863921 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:58.864045 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:58.864781 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:58.864787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:58.865139 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:58.865239 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.865245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.866379 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.866382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.867305 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:58.867315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.868086 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.868090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:58.868192 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.868728 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:58.868739 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:58.870179 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:58.872183 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:58.872190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:58.872194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:58.872462 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:58.872756 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:58.872758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:58.872760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:58.872834 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:58.873295 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:58.873300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:58.873458 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 00:50:58.873593 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 00:50:58.873735 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":109,"threads_started":1}
I20260925 00:50:58.874034 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:58.863363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.0.249.65:35061)
0925 00:50:58.864307 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:58.864332 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:58.864374 (+ 42us) server_negotiation.cc:407] Connection header received
0925 00:50:58.864441 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:58.864447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:58.864651 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:58.864780 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:58.865364 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.866199 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.867453 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:58.867921 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:58.868065 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:58.870311 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:58.870327 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:58.870340 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:58.870364 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:58.872068 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:58.872571 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:58.872577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:58.872579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:58.872668 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:58.872923 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:58.872926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:58.872928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:58.873192 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:58.873315 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:58.873695 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 00:50:58.873885 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":108,"threads_started":1}
I20260925 00:50:58.875192 15436 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c57a3545704b70827ab8365f047e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:58 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:52792
I20260925 00:50:58.876001 15511 tablet_replica.cc:330] stopping tablet replica
I20260925 00:50:58.876283 15511 raft_consensus.cc:2229] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:50:58.876526 15511 raft_consensus.cc:2258] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:50:58.877511 15511 ts_tablet_manager.cc:1905] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:50:58.880761 15511 ts_tablet_manager.cc:1918] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:50:58.880839 15511 log.cc:1196] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/a6c57a3545704b70827ab8365f047e87
I20260925 00:50:58.881151 15511 ts_tablet_manager.cc:1939] T a6c57a3545704b70827ab8365f047e87 P 604494106bcd4573aabbfb1b93b656e2: Deleting consensus metadata
I20260925 00:50:58.882738 15298 catalog_manager.cc:4841] TS 604494106bcd4573aabbfb1b93b656e2 (127.0.249.65:35061): tablet a6c57a3545704b70827ab8365f047e87 (table test-table [id=89010e5037e945e2989ccdec9834ee0d]) successfully deleted
W20260925 00:50:58.955351 15506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:58.955634 15506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:58.955710 15506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:58.959986 15506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:58.960071 15506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:58.960155 15506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:50:58.965687 15506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:40617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=32779
--tserver_master_addrs=127.0.249.126:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:58.966676 15506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:58.967643 15506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:58.976886 15516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:58.976874 15519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:58.976874 15517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:58.976991 15506 server_base.cc:1034] running on GCE node
I20260925 00:50:58.977527 15506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:58.978111 15506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:58.979282 15506 hybrid_clock.cc:648] HybridClock initialized: now 1790297458979259 us; error 48 us; skew 500 ppm
Sep 25 00:50:58 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:58.981884 15506 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:50:58.982645 15506 webserver.cc:460] Webserver started at http://127.0.249.66:32779/ using document root <none> and password file <none>
I20260925 00:50:58.983212 15506 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:58.983314 15506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:58.987355 15506 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:50:58.989212 15526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:58.990420 15506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:50:58.990540 15506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "8f80559ea0ce4b1ba9f4a996726e7392"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "9c344ff38fa5e80fec72cce042d7af5f"
server_key_iv: "bcc7172639a7ec8a87779ba188283ce4"
server_key_version: "encryptionkey@0"
I20260925 00:50:58.991052 15506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:59.004387 15506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:59.007157 15506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:59.007351 15506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:59.007918 15506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:59.008726 15506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:59.008771 15506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:59.008839 15506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:59.008885 15506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:59.017834 15506 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:40617
I20260925 00:50:59.017884 15641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:40617 every 8 connection(s)
I20260925 00:50:59.018802 15506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:50:59.026542 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 15506
I20260925 00:50:59.026940 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:42289
--local_ip_for_outbound_sockets=127.0.249.67
--tserver_master_addrs=127.0.249.126:36105
--webserver_port=42835
--webserver_interface=127.0.249.67
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:50:59 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:59.032965 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.020583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:46277 (local address 127.0.249.126:36105)
0925 00:50:59.020754 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:59.020759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:59.021611 (+ 852us) server_negotiation.cc:407] Connection header received
0925 00:50:59.023004 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:59.023008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:59.023063 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:59.023155 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:59.024353 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.024949 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.026190 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.026455 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.026579 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.029135 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:59.029139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:59.029144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:59.029180 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:59.030780 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:59.031319 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:59.031325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:59.031327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:59.031415 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:59.031786 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:59.031789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:59.031791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:59.032015 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:59.032130 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:59.032631 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 00:50:59.032783 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:50:59.033826 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.020958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:46277)
0925 00:50:59.021448 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.021505 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:59.022784 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:59.023348 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:59.023355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:59.023715 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:59.024191 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.024205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.025087 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.025092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.026046 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.026056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.026614 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.026619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.026811 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.027442 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:59.027459 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:59.028959 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:59.030908 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:59.030914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:59.030929 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:59.031207 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:59.031517 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:59.031522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:59.031524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:59.031674 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:59.032128 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:59.032135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:59.032386 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 00:50:59.032585 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 00:50:59.032916 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260925 00:50:59.035044 15642 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36105
I20260925 00:50:59.035310 15642 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:59.035799 15642 heartbeater.cc:511] Master 127.0.249.126:36105 requested a full tablet report, sending...
I20260925 00:50:59.036908 15314 ts_manager.cc:194] Registered new tserver with Master: 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617)
I20260925 00:50:59.037699 15314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:46277
W20260925 00:50:59.137213 15645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:59.137480 15645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:59.137521 15645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:59.141312 15645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:59.141363 15645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:59.141449 15645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:50:59.146513 15645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:42289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=42835
--tserver_master_addrs=127.0.249.126:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:59.147460 15645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:59.148386 15645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:59.157840 15650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:59.157898 15651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:59.157840 15653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:59.158008 15645 server_base.cc:1034] running on GCE node
I20260925 00:50:59.158430 15645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:59.159068 15645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:59.160224 15645 hybrid_clock.cc:648] HybridClock initialized: now 1790297459160181 us; error 54 us; skew 500 ppm
Sep 25 00:50:59 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:59.163183 15645 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:50:59.164179 15645 webserver.cc:460] Webserver started at http://127.0.249.67:42835/ using document root <none> and password file <none>
I20260925 00:50:59.164750 15645 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:59.164856 15645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:59.168576 15645 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:50:59.170687 15660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:59.171855 15645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:50:59.171990 15645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "9fda2bc1e81248d7ba08065a7190b50f"
format_stamp: "Formatted at 2026-09-25 00:50:58 on dist-test-slave-05m0"
server_key: "077dc667c09369cab021b7f800fc4383"
server_key_iv: "c28b6c2c1a86602c318d542eca513401"
server_key_version: "encryptionkey@0"
I20260925 00:50:59.172534 15645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:59.183394 15645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:59.186851 15645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:59.187055 15645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:59.187636 15645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:59.188445 15645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:50:59.188515 15645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:59.188586 15645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:50:59.188638 15645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:59.197666 15645 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:42289
I20260925 00:50:59.197693 15773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:42289 every 8 connection(s)
I20260925 00:50:59.198621 15645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:50:59.203527 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 15645
Sep 25 00:50:59 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:50:59.212616 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.204208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.0.249.126:36105)
0925 00:50:59.204453 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:59.204459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:59.204690 (+ 231us) server_negotiation.cc:407] Connection header received
0925 00:50:59.204859 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:59.204862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:59.204910 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:59.205002 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:59.205803 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.206312 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.207336 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.207679 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.207789 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.209176 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:59.209179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:59.209182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:59.209220 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:59.210865 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:59.211315 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:59.211318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:59.211320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:59.211400 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:59.211674 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:59.211677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:59.211678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:59.211980 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:59.212176 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:59.212346 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 00:50:59.212446 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":171,"spinlock_wait_cycles":14976,"thread_start_us":68,"threads_started":1}
I20260925 00:50:59.213076 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.200354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:44079 (local address 127.0.249.126:36105)
0925 00:50:59.200486 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:59.200490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:59.201313 (+ 823us) server_negotiation.cc:407] Connection header received
0925 00:50:59.202598 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:59.202601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:59.202657 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:59.202739 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:50:59.203916 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.204567 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.205773 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.206006 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.206114 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.208665 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:50:59.208673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:50:59.208675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:50:59.208698 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:50:59.210603 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:59.211179 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:59.211181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:59.211182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:59.211252 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:50:59.211674 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:50:59.211677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:50:59.211678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:50:59.211980 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:50:59.212081 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:59.212789 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 00:50:59.212907 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":49,"spinlock_wait_cycles":4608}
I20260925 00:50:59.214442 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.200744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:44079)
0925 00:50:59.201166 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.201207 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:59.202394 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:59.202911 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:59.202918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:59.203251 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:50:59.203772 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.203787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.204691 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.204694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.205655 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.205664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.206150 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.206154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.206321 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.206900 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:50:59.206916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:50:59.208501 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:50:59.210725 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:59.210730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:59.210742 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:59.211082 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:59.211380 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:50:59.211382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:50:59.211384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:50:59.211578 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:50:59.212093 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:50:59.212099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:50:59.212442 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 00:50:59.213265 (+ 823us) client_negotiation.cc:240] Negotiation successful
0925 00:50:59.213638 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260925 00:50:59.215512 15774 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36105
I20260925 00:50:59.215745 15774 heartbeater.cc:464] Registering TS with master...
I20260925 00:50:59.216178 15774 heartbeater.cc:511] Master 127.0.249.126:36105 requested a full tablet report, sending...
I20260925 00:50:59.217106 15314 ts_manager.cc:194] Registered new tserver with Master: 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289)
I20260925 00:50:59.217689 15314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:44079
I20260925 00:50:59.226130 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:50:59.230852 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.226701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.0.249.126:36105)
0925 00:50:59.226853 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:59.226860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:59.227082 (+ 222us) server_negotiation.cc:407] Connection header received
0925 00:50:59.227271 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:59.227274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:59.227372 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:59.227481 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:50:59.227729 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.228233 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.229504 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.229824 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.229911 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.230179 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:50:59.230562 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:50:59.230651 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:59.230706 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 00:50:59.230753 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:50:59.231467 15314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:50:59.234731 15314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:50:59.242244 15436 tablet_service.cc:1467] Processing CreateTablet for tablet e4d98164a8124253aca2eeeadc901bd5 (DEFAULT_TABLE table=test-table [id=8779d6feaa1e46cd82d1ab884e038592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:59.242871 15436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d98164a8124253aca2eeeadc901bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:59.245755 15394 tablet_bootstrap.cc:492] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Bootstrap starting.
I20260925 00:50:59.246620 15394 tablet_bootstrap.cc:654] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:59.247135 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.241182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42289 (local address 127.0.0.1:57164)
0925 00:50:59.241314 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.241331 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:59.241431 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:59.243566 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:59.243569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:59.243587 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:59.243669 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.243674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.244785 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.244791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.245667 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.245682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.246703 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.246706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.246799 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.246851 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 00:50:59.246925 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 00:50:59.247010 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 00:50:59.247184 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.241431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.0.249.67:42289)
0925 00:50:59.243032 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:59.243045 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:59.243065 (+ 20us) server_negotiation.cc:407] Connection header received
0925 00:50:59.243134 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:59.243139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:59.243436 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:59.243590 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:59.243764 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.244659 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.245774 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.246580 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.246704 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.246853 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:59.246964 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 00:50:59.247056 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":187,"threads_started":1}
I20260925 00:50:59.247982 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.241429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:40617 (local address 127.0.0.1:57676)
0925 00:50:59.241896 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.241907 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:59.241974 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:59.242651 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:59.242654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:59.242671 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:59.242806 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.242824 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.244230 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.244233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.245542 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.245554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.247585 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.247589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.247677 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.247708 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 00:50:59.247771 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:50:59.247855 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":119,"threads_started":1}
I20260925 00:50:59.248217 15394 tablet_bootstrap.cc:492] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: No bootstrap required, opened a new log
I20260925 00:50:59.248308 15394 ts_tablet_manager.cc:1397] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 00:50:59.248337 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.241862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.0.249.66:40617)
0925 00:50:59.242189 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:59.242199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:59.242218 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:50:59.242281 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:59.242285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:59.242502 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:59.242676 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:59.242951 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.244089 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.245916 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.246883 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.247010 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.247181 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:59.248091 (+ 910us) server_negotiation.cc:299] Negotiation successful
0925 00:50:59.248189 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":102,"threads_started":1}
I20260925 00:50:59.248746 15394 raft_consensus.cc:348] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.248867 15394 raft_consensus.cc:374] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:59.248903 15394 raft_consensus.cc:729] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604494106bcd4573aabbfb1b93b656e2, State: Initialized, Role: FOLLOWER
I20260925 00:50:59.249043 15394 consensus_queue.cc:260] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.249408 15394 ts_tablet_manager.cc:1428] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:50:59.250407 15708 tablet_service.cc:1467] Processing CreateTablet for tablet e4d98164a8124253aca2eeeadc901bd5 (DEFAULT_TABLE table=test-table [id=8779d6feaa1e46cd82d1ab884e038592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:59.251447 15708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d98164a8124253aca2eeeadc901bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:59.251406 15574 tablet_service.cc:1467] Processing CreateTablet for tablet e4d98164a8124253aca2eeeadc901bd5 (DEFAULT_TABLE table=test-table [id=8779d6feaa1e46cd82d1ab884e038592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:50:59.252498 15574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d98164a8124253aca2eeeadc901bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:50:59.258582 15787 tablet_bootstrap.cc:492] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: Bootstrap starting.
I20260925 00:50:59.259197 15788 tablet_bootstrap.cc:492] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: Bootstrap starting.
I20260925 00:50:59.260732 15787 tablet_bootstrap.cc:654] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:59.261494 15788 tablet_bootstrap.cc:654] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: Neither blocks nor log segments found. Creating new log.
I20260925 00:50:59.261854 15787 log.cc:824] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:59.262238 15788 log.cc:824] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:59.263846 15787 tablet_bootstrap.cc:492] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: No bootstrap required, opened a new log
I20260925 00:50:59.264016 15788 tablet_bootstrap.cc:492] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: No bootstrap required, opened a new log
I20260925 00:50:59.264055 15787 ts_tablet_manager.cc:1397] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:50:59.264214 15788 ts_tablet_manager.cc:1397] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:50:59.267184 15787 raft_consensus.cc:348] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.267184 15788 raft_consensus.cc:348] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.267403 15788 raft_consensus.cc:374] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:59.267433 15787 raft_consensus.cc:374] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:50:59.267467 15788 raft_consensus.cc:729] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f80559ea0ce4b1ba9f4a996726e7392, State: Initialized, Role: FOLLOWER
I20260925 00:50:59.267518 15787 raft_consensus.cc:729] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fda2bc1e81248d7ba08065a7190b50f, State: Initialized, Role: FOLLOWER
I20260925 00:50:59.267908 15788 consensus_queue.cc:260] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.267913 15787 consensus_queue.cc:260] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.268551 15774 heartbeater.cc:503] Master 127.0.249.126:36105 was elected leader, sending a full tablet report...
I20260925 00:50:59.268777 15642 heartbeater.cc:503] Master 127.0.249.126:36105 was elected leader, sending a full tablet report...
I20260925 00:50:59.268863 15787 ts_tablet_manager.cc:1428] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:50:59.268858 15788 ts_tablet_manager.cc:1428] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 00:50:59.269624 15643 tablet.cc:2369] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:50:59.336670 15503 tablet.cc:2369] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:50:59.384480 15792 raft_consensus.cc:482] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:50:59.384708 15792 raft_consensus.cc:504] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.386016 15792 leader_election.cc:288] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289), 604494106bcd4573aabbfb1b93b656e2 (127.0.249.65:35061)
I20260925 00:50:59.389742 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.386275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42289 (local address 127.0.249.66:58423)
0925 00:50:59.386409 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.386430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:59.386548 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:59.386833 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:59.386836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:59.386857 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:59.386981 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.386988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.387711 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.387717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.388718 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.388728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.389367 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.389370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.389504 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.389524 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:50:59.389583 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 00:50:59.389641 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 00:50:59.389812 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.386337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:58423 (local address 127.0.249.67:42289)
0925 00:50:59.386480 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:59.386487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:59.386505 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:50:59.386663 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:59.386665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:59.386709 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:59.386815 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:59.387108 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.387604 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.388849 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.389240 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.389356 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.389442 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:59.389619 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 00:50:59.389693 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:50:59.390446 15728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d98164a8124253aca2eeeadc901bd5" candidate_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fda2bc1e81248d7ba08065a7190b50f" is_pre_election: true
I20260925 00:50:59.390604 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.386275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:35061 (local address 127.0.249.66:40889)
0925 00:50:59.387060 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.387076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:59.387198 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:59.387518 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:50:59.387520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:50:59.387539 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:50:59.387680 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.387687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.388384 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.388387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.389258 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:50:59.389266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.390197 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.390201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:50:59.390292 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.390314 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:50:59.390403 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 00:50:59.390472 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":360,"thread_start_us":88,"threads_started":1}
I20260925 00:50:59.390618 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.386337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:40889 (local address 127.0.249.65:35061)
0925 00:50:59.386604 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:59.386610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:59.387159 (+ 549us) server_negotiation.cc:407] Connection header received
0925 00:50:59.387313 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:59.387316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:59.387386 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:59.387495 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:50:59.387798 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.388274 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.389419 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.390047 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.390183 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.390287 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:59.390429 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 00:50:59.390490 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260925 00:50:59.390769 15728 raft_consensus.cc:2454] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f80559ea0ce4b1ba9f4a996726e7392 in term 0.
I20260925 00:50:59.391255 15530 leader_election.cc:302] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f80559ea0ce4b1ba9f4a996726e7392, 9fda2bc1e81248d7ba08065a7190b50f; no voters:
I20260925 00:50:59.391238 15456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d98164a8124253aca2eeeadc901bd5" candidate_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604494106bcd4573aabbfb1b93b656e2" is_pre_election: true
I20260925 00:50:59.391516 15456 raft_consensus.cc:2454] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f80559ea0ce4b1ba9f4a996726e7392 in term 0.
I20260925 00:50:59.391533 15792 raft_consensus.cc:2781] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:50:59.391614 15792 raft_consensus.cc:482] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:50:59.391659 15792 raft_consensus.cc:3037] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:59.392882 15792 raft_consensus.cc:504] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.393183 15792 leader_election.cc:288] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [CANDIDATE]: Term 1 election: Requested vote from peers 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289), 604494106bcd4573aabbfb1b93b656e2 (127.0.249.65:35061)
I20260925 00:50:59.393528 15728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d98164a8124253aca2eeeadc901bd5" candidate_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fda2bc1e81248d7ba08065a7190b50f"
I20260925 00:50:59.393527 15456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d98164a8124253aca2eeeadc901bd5" candidate_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604494106bcd4573aabbfb1b93b656e2"
I20260925 00:50:59.393627 15728 raft_consensus.cc:3037] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:59.393656 15456 raft_consensus.cc:3037] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:50:59.394749 15728 raft_consensus.cc:2454] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f80559ea0ce4b1ba9f4a996726e7392 in term 1.
I20260925 00:50:59.394785 15456 raft_consensus.cc:2454] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f80559ea0ce4b1ba9f4a996726e7392 in term 1.
I20260925 00:50:59.395110 15527 leader_election.cc:302] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604494106bcd4573aabbfb1b93b656e2, 8f80559ea0ce4b1ba9f4a996726e7392; no voters:
I20260925 00:50:59.395306 15792 raft_consensus.cc:2781] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:50:59.395531 15792 raft_consensus.cc:686] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 LEADER]: Becoming Leader. State: Replica: 8f80559ea0ce4b1ba9f4a996726e7392, State: Running, Role: LEADER
I20260925 00:50:59.395871 15792 consensus_queue.cc:237] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.399097 15314 catalog_manager.cc:5495] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66). New cstate: current_term: 1 leader_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } health_report { overall_health: UNKNOWN } } }
I20260925 00:50:59.437319 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.434169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.0.249.66:40617)
0925 00:50:59.434291 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 00:50:59.434295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:50:59.434308 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:50:59.434441 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:50:59.434443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:50:59.434497 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:50:59.434585 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:50:59.434796 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.435281 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.436067 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:50:59.436275 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:50:59.436349 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:50:59.436606 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:50:59.437018 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:50:59.437105 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:50:59.437165 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 00:50:59.437213 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:50:59.443085 15456 raft_consensus.cc:1261] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Refusing update from remote peer 8f80559ea0ce4b1ba9f4a996726e7392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:59.443090 15728 raft_consensus.cc:1261] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [term 1 FOLLOWER]: Refusing update from remote peer 8f80559ea0ce4b1ba9f4a996726e7392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:50:59.443830 15792 consensus_queue.cc:1035] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:50:59.444084 15795 consensus_queue.cc:1035] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:50:59.449151 15797 mvcc.cc:205] Tried to move back new op lower bound from 7333058393871519744 to 7333058393688227840. Current Snapshot: MvccSnapshot[applied={T|T < 7333058393871519744}]
W20260925 00:50:59.449800 15775 tablet.cc:2369] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:50:59.451831 15798 mvcc.cc:205] Tried to move back new op lower bound from 7333058393871519744 to 7333058393688227840. Current Snapshot: MvccSnapshot[applied={T|T < 7333058393871519744 or (T in {7333058393871519744})}]
I20260925 00:50:59.452280 15799 mvcc.cc:205] Tried to move back new op lower bound from 7333058393871519744 to 7333058393688227840. Current Snapshot: MvccSnapshot[applied={T|T < 7333058393871519744}]
I20260925 00:50:59.458699 15313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:50:59.458832 15313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:50:59.461022 15313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Sending DeleteTablet for 3 replicas of tablet e4d98164a8124253aca2eeeadc901bd5
I20260925 00:50:59.461725 15436 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d98164a8124253aca2eeeadc901bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:59 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:52792
I20260925 00:50:59.461795 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 15352
I20260925 00:50:59.462400 15708 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d98164a8124253aca2eeeadc901bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:59 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:57164
I20260925 00:50:59.462946 15806 tablet_replica.cc:330] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: stopping tablet replica
I20260925 00:50:59.463306 15806 raft_consensus.cc:2229] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:50:59.463678 15806 raft_consensus.cc:2258] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:50:59.463629 15574 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d98164a8124253aca2eeeadc901bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:50:59 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:57676
I20260925 00:50:59.464054 15807 tablet_replica.cc:330] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: stopping tablet replica
I20260925 00:50:59.464341 15807 raft_consensus.cc:2229] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:50:59.464731 15807 raft_consensus.cc:2258] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:50:59.465426 15806 ts_tablet_manager.cc:1905] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:50:59.466078 15807 ts_tablet_manager.cc:1905] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:50:59.468227 15806 ts_tablet_manager.cc:1918] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:50:59.468400 15806 log.cc:1196] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/e4d98164a8124253aca2eeeadc901bd5
I20260925 00:50:59.468807 15806 ts_tablet_manager.cc:1939] T e4d98164a8124253aca2eeeadc901bd5 P 9fda2bc1e81248d7ba08065a7190b50f: Deleting consensus metadata
I20260925 00:50:59.469523 15807 ts_tablet_manager.cc:1918] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:50:59.469622 15807 log.cc:1196] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/e4d98164a8124253aca2eeeadc901bd5
I20260925 00:50:59.469969 15807 ts_tablet_manager.cc:1939] T e4d98164a8124253aca2eeeadc901bd5 P 8f80559ea0ce4b1ba9f4a996726e7392: Deleting consensus metadata
I20260925 00:50:59.470216 15301 catalog_manager.cc:4841] TS 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289): tablet e4d98164a8124253aca2eeeadc901bd5 (table test-table [id=8779d6feaa1e46cd82d1ab884e038592]) successfully deleted
W20260925 00:50:59.470832 15298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:35061 (error 108)
I20260925 00:50:59.471014 15301 catalog_manager.cc:4841] TS 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617): tablet e4d98164a8124253aca2eeeadc901bd5 (table test-table [id=8779d6feaa1e46cd82d1ab884e038592]) successfully deleted
I20260925 00:50:59.471113 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 15506
I20260925 00:50:59.472584 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.472253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:35061 (local address 127.0.0.1:52794)
0925 00:50:59.472402 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.472482 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 00:50:59.472824 15298 catalog_manager.cc:4584] TS 604494106bcd4573aabbfb1b93b656e2 (127.0.249.65:35061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4d98164a8124253aca2eeeadc901bd5: Network error: Client connection negotiation failed: client connection to 127.0.249.65:35061: connect: Connection refused (error 111)
I20260925 00:50:59.477239 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 15645
I20260925 00:50:59.483484 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 15281
I20260925 00:50:59.490168 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36105
--webserver_interface=127.0.249.126
--webserver_port=40813
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:36105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:50:59.599241 15809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:59.599529 15809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:59.599570 15809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:59.603615 15809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:50:59.603681 15809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:59.603700 15809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:59.603717 15809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:50:59.603734 15809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:50:59.609135 15809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:36105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=40813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:59.610246 15809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:59.611212 15809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:59.619544 15817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:59.619580 15814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:59.619573 15815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:59.619956 15809 server_base.cc:1034] running on GCE node
I20260925 00:50:59.620496 15809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:59.621701 15809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:59.622860 15809 hybrid_clock.cc:648] HybridClock initialized: now 1790297459622821 us; error 51 us; skew 500 ppm
Sep 25 00:50:59 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:59.625682 15809 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:50:59.626549 15809 webserver.cc:460] Webserver started at http://127.0.249.126:40813/ using document root <none> and password file <none>
I20260925 00:50:59.627120 15809 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:59.627213 15809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:59.630963 15809 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 00:50:59.633040 15824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:59.634258 15809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:50:59.634406 15809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "95da1db27d0e4209acf733e267888294"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "891a017b05ba7e28dec331e71487dc0b"
server_key_iv: "06934c921697bcfb3fdb310e0ef99392"
server_key_version: "encryptionkey@0"
I20260925 00:50:59.634974 15809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:59.654055 15809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:59.657624 15809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:59.657848 15809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:59.664714 15809 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:36105
I20260925 00:50:59.665022 15877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:36105 every 8 connection(s)
I20260925 00:50:59.665803 15809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:50:59.666350 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 15809
I20260925 00:50:59.666828 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:35061
--local_ip_for_outbound_sockets=127.0.249.65
--tserver_master_addrs=127.0.249.126:36105
--webserver_port=37631
--webserver_interface=127.0.249.65
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:50:59.670946 15878 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:50:59.672510 15878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Bootstrap starting.
I20260925 00:50:59.689252 15878 log.cc:824] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:59.699949 15878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:50:59.700327 15878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294: Bootstrap complete.
I20260925 00:50:59.704238 15878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:59.704620 15878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95da1db27d0e4209acf733e267888294, State: Initialized, Role: FOLLOWER
I20260925 00:50:59.705143 15878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:59.705262 15878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:50:59.705345 15878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:50:59.705476 15878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 2 FOLLOWER]: Advancing to term 3
I20260925 00:50:59.707278 15878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:59.707597 15878 leader_election.cc:302] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95da1db27d0e4209acf733e267888294; no voters:
I20260925 00:50:59.707912 15878 leader_election.cc:288] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 00:50:59.707993 15882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 00:50:59.708186 15882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [term 3 LEADER]: Becoming Leader. State: Replica: 95da1db27d0e4209acf733e267888294, State: Running, Role: LEADER
I20260925 00:50:59.708479 15882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } }
I20260925 00:50:59.709147 15878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:50:59.709820 15883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "95da1db27d0e4209acf733e267888294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } } }
I20260925 00:50:59.709954 15883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:59.710528 15884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95da1db27d0e4209acf733e267888294. Latest consensus state: current_term: 3 leader_uuid: "95da1db27d0e4209acf733e267888294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1db27d0e4209acf733e267888294" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36105 } } }
I20260925 00:50:59.710675 15884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da1db27d0e4209acf733e267888294 [sys.catalog]: This master's current role is: LEADER
I20260925 00:50:59.710911 15889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:50:59.715857 15889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:50:59.716738 15889 catalog_manager.cc:1259] Loaded cluster ID: 7ff1d788dadb4e7f9e5b18ec26aecc5a
I20260925 00:50:59.716799 15889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 00:50:59.717769 15889 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc9de5a59ad google::LogMessage::Fail() at ??:0
@ 0x7fc9de5a888a google::LogMessage::SendToLog() at ??:0
@ 0x7fc9de5a54de google::LogMessage::Flush() at ??:0
@ 0x7fc9de5a6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc9e8ee4fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc9e8ee5d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc9e8ee2579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc9e8f26640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc9e07f73d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc9e07f7c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc9e07f93f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc9e07e8512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc9e1e2c6db start_thread at ??:0
@ 0x7fc9dd0a361f clone at ??:0
W20260925 00:50:59.810703 15880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:50:59.811072 15880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:50:59.811143 15880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:50:59.817377 15880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:50:59.817482 15880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:50:59.817572 15880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:50:59.825434 15880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:35061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=37631
--tserver_master_addrs=127.0.249.126:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:50:59.826862 15880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:50:59.828205 15880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:50:59.839047 15920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:59.839068 15919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:50:59.839066 15922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:50:59.839403 15880 server_base.cc:1034] running on GCE node
I20260925 00:50:59.839819 15880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:50:59.840482 15880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:50:59.841648 15880 hybrid_clock.cc:648] HybridClock initialized: now 1790297459841619 us; error 46 us; skew 500 ppm
Sep 25 00:50:59 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:50:59.844388 15880 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:50:59.845261 15880 webserver.cc:460] Webserver started at http://127.0.249.65:37631/ using document root <none> and password file <none>
I20260925 00:50:59.845836 15880 fs_manager.cc:359] Metadata directory not provided
I20260925 00:50:59.845901 15880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:50:59.849684 15880 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 00:50:59.851907 15929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:50:59.853204 15880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:50:59.853338 15880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "604494106bcd4573aabbfb1b93b656e2"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "87ed42c0b031b86cab56017a5bc7c06c"
server_key_iv: "56f7c97cc5f8790c43b8904c7992a50f"
server_key_version: "encryptionkey@0"
I20260925 00:50:59.853940 15880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:50:59.887645 15880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:50:59.890920 15880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:50:59.891134 15880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:50:59.891743 15880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:50:59.892899 15936 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:50:59.898224 15880 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:50:59.898298 15880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 00:50:59.898389 15880 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:50:59.900023 15880 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:50:59.900094 15880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 00:50:59.900373 15936 tablet_bootstrap.cc:492] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Bootstrap starting.
I20260925 00:50:59.910848 15880 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:35061
I20260925 00:50:59.910912 16043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:35061 every 8 connection(s)
I20260925 00:50:59.912091 15880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:50:59.914247 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 15880
I20260925 00:50:59.914714 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:40617
--local_ip_for_outbound_sockets=127.0.249.66
--tserver_master_addrs=127.0.249.126:36105
--webserver_port=32779
--webserver_interface=127.0.249.66
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:50:59.920711 15936 log.cc:824] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Log is configured to *not* fsync() on all Append() calls
I20260925 00:50:59.927804 15936 tablet_bootstrap.cc:492] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:50:59.928210 15936 tablet_bootstrap.cc:492] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Bootstrap complete.
I20260925 00:50:59.929426 15936 ts_tablet_manager.cc:1397] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.011s
I20260925 00:50:59.931846 15936 raft_consensus.cc:348] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.932286 15936 raft_consensus.cc:729] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604494106bcd4573aabbfb1b93b656e2, State: Initialized, Role: FOLLOWER
I20260925 00:50:59.932802 15936 consensus_queue.cc:260] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:50:59.933810 15936 ts_tablet_manager.cc:1428] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260925 00:50:59.952732 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:50:59.914161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:50373)
0925 00:50:59.914807 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.914891 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 00:50:59.916127 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:50:59.951952 (+ 35825us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":158,"threads_started":1}
W20260925 00:50:59.953419 16044 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:36105 (0 consecutive failures): Network error: Failed to ping master at 127.0.249.126:36105: Client connection negotiation failed: client connection to 127.0.249.126:36105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 00:50:59.954214 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.953923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:47521)
0925 00:50:59.954058 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.954157 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 00:50:59.954957 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:50:59.954741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:39285)
0925 00:50:59.954849 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 00:50:59.954896 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 00:50:59.955263 16044 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:51:00.036302 16047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:00.036548 16047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:00.036633 16047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:00.040606 16047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:00.040665 16047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:00.040766 16047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:51:00.046087 16047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:40617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=32779
--tserver_master_addrs=127.0.249.126:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:00.047123 16047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:00.048126 16047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:00.057802 16057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:00.057825 16055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:00.057793 16054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:00.058138 16047 server_base.cc:1034] running on GCE node
I20260925 00:51:00.058684 16047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:00.059266 16047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:00.060453 16047 hybrid_clock.cc:648] HybridClock initialized: now 1790297460060436 us; error 26 us; skew 500 ppm
Sep 25 00:51:00 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:00.063243 16047 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:51:00.064191 16047 webserver.cc:460] Webserver started at http://127.0.249.66:32779/ using document root <none> and password file <none>
I20260925 00:51:00.064776 16047 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:00.064872 16047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:00.068643 16047 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:51:00.070691 16064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:00.071940 16047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:51:00.072109 16047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "8f80559ea0ce4b1ba9f4a996726e7392"
format_stamp: "Formatted at 2026-09-25 00:50:57 on dist-test-slave-05m0"
server_key: "9c344ff38fa5e80fec72cce042d7af5f"
server_key_iv: "bcc7172639a7ec8a87779ba188283ce4"
server_key_version: "encryptionkey@0"
I20260925 00:51:00.072642 16047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:00.094452 16047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:00.097259 16047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:00.097489 16047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:00.098093 16047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:00.098935 16047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:51:00.099005 16047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:00.099072 16047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:51:00.099123 16047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:00.108748 16047 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:40617
I20260925 00:51:00.108767 16177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:40617 every 8 connection(s)
I20260925 00:51:00.109808 16047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:51:00.112222 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 16047
I20260925 00:51:00.112653 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:42289
--local_ip_for_outbound_sockets=127.0.249.67
--tserver_master_addrs=127.0.249.126:36105
--webserver_port=42835
--webserver_interface=127.0.249.67
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:51:00.113662 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:00.111860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:43419)
0925 00:51:00.112356 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:00.112894 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":151,"threads_started":1}
W20260925 00:51:00.114117 16178 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:36105 (0 consecutive failures): Network error: Failed to ping master at 127.0.249.126:36105: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
I20260925 00:51:00.114718 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:00.114501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:44551)
0925 00:51:00.114615 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:00.114668 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 00:51:00.115347 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:00.115116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:57463)
0925 00:51:00.115219 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:00.115276 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 00:51:00.116571 16178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:51:00.223266 16181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:00.223552 16181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:00.223599 16181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:00.227604 16181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:00.227659 16181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:00.227716 16181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:51:00.232962 16181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:42289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=42835
--tserver_master_addrs=127.0.249.126:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:00.233960 16181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:00.234912 16181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:00.244745 16189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:00.244792 16186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:00.244772 16187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:00.245110 16181 server_base.cc:1034] running on GCE node
I20260925 00:51:00.245687 16181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:00.246366 16181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:00.247577 16181 hybrid_clock.cc:648] HybridClock initialized: now 1790297460247561 us; error 25 us; skew 500 ppm
Sep 25 00:51:00 dist-test-slave-05m0 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:00.250552 16181 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:51:00.251497 16181 webserver.cc:460] Webserver started at http://127.0.249.67:42835/ using document root <none> and password file <none>
I20260925 00:51:00.252063 16181 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:00.252154 16181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:00.256374 16181 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:51:00.258596 16196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:00.259930 16181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:51:00.260062 16181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "9fda2bc1e81248d7ba08065a7190b50f"
format_stamp: "Formatted at 2026-09-25 00:50:58 on dist-test-slave-05m0"
server_key: "077dc667c09369cab021b7f800fc4383"
server_key_iv: "c28b6c2c1a86602c318d542eca513401"
server_key_version: "encryptionkey@0"
I20260925 00:51:00.260604 16181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:00.300649 16181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:00.304211 16181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:00.304442 16181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:00.305050 16181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:00.305898 16181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:51:00.305969 16181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:00.306039 16181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:51:00.306109 16181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:00.315532 16181 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:42289
I20260925 00:51:00.315572 16309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:42289 every 8 connection(s)
I20260925 00:51:00.316530 16181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:51:00.320179 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 16181
I20260925 00:51:00.320305 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:00.318710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:40277)
0925 00:51:00.319154 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:00.319604 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":96,"threads_started":1}
W20260925 00:51:00.320828 16310 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:36105 (0 consecutive failures): Network error: Failed to ping master at 127.0.249.126:36105: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
I20260925 00:51:00.321468 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:00.321163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:56639)
0925 00:51:00.321325 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:00.321374 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":65}
I20260925 00:51:00.322078 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:00.321870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:56545)
0925 00:51:00.321977 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:00.322020 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 00:51:00.322278 16310 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 00:51:00.956239 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:00.955831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:46143)
0925 00:51:00.956071 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:00.956125 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
I20260925 00:51:01.077458 16315 raft_consensus.cc:482] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:51:01.077718 16315 raft_consensus.cc:504] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:51:01.078815 16315 leader_election.cc:288] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289), 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617)
Sep 25 00:51:01 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.66@KRBTEST.COM
Sep 25 00:51:01 dist-test-slave-05m0 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.67@KRBTEST.COM
I20260925 00:51:01.095742 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:01.078992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42289 (local address 127.0.249.65:47271)
0925 00:51:01.079131 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:01.079145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:01.079264 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:01.081452 (+ 2188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:01.081459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:01.082056 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:01.082966 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:01.082986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:01.084885 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:01.084889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:01.086484 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:01.086494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:01.087447 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:01.087450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:01.087743 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:01.088446 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:01.088469 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:01.090712 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:01.092873 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:01.092879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:01.092893 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:01.093410 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:01.093786 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:01.093790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:01.093792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:01.094004 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:01.094738 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:01.094746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:01.095048 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 00:51:01.095248 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 00:51:01.095552 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":417,"spinlock_wait_cycles":11392}
I20260925 00:51:01.095920 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:01.079260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:40617 (local address 127.0.249.65:38223)
0925 00:51:01.079506 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:01.079518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:01.079593 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:01.081452 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:01.081458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:01.082056 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:01.082966 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:01.082986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:01.084885 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:01.084889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:01.086485 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:01.086494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:01.087434 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:01.087437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:01.087740 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:01.088477 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:01.088488 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:01.090400 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:01.092873 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:01.092879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:01.092893 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:01.093405 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:01.093786 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:01.093790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:01.093792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:01.094004 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:01.095335 (+ 1331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:01.095339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:01.095432 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 00:51:01.095531 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 00:51:01.095657 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":326,"spinlock_wait_cycles":43136,"thread_start_us":49,"threads_started":1}
I20260925 00:51:01.096168 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:01.079429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:38223 (local address 127.0.249.66:40617)
0925 00:51:01.079767 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:01.079779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:01.079809 (+ 30us) server_negotiation.cc:407] Connection header received
0925 00:51:01.080520 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:01.080529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:01.080998 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:01.081329 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:01.083149 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:01.084721 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:01.086604 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:01.087311 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:01.087494 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:01.090610 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:01.090615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:01.090627 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:01.090653 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:01.092740 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:01.093515 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:01.093522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:01.093528 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:01.093688 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:01.094779 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:01.094782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:01.094784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:01.095230 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:01.095359 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:01.095697 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 00:51:01.095970 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":59,"threads_started":1}
I20260925 00:51:01.096431 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:01.079122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:47271 (local address 127.0.249.67:42289)
0925 00:51:01.079555 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:01.079572 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:01.079616 (+ 44us) server_negotiation.cc:407] Connection header received
0925 00:51:01.080520 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:01.080529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:01.080998 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:01.081329 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:01.083149 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:01.084721 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:01.086604 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:01.087311 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:01.087494 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:01.090896 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:01.090904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:01.090920 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:01.090955 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:01.092754 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:01.093515 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:01.093522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:01.093528 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:01.093688 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:01.094115 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:01.094118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:01.094120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:01.094568 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:01.094720 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:01.095423 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 00:51:01.095706 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":94,"threads_started":1}
W20260925 00:51:01.098618 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
W20260925 00:51:01.098852 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
I20260925 00:51:01.098954 15933 leader_election.cc:302] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 604494106bcd4573aabbfb1b93b656e2; no voters: 8f80559ea0ce4b1ba9f4a996726e7392, 9fda2bc1e81248d7ba08065a7190b50f
I20260925 00:51:01.099216 16315 raft_consensus.cc:2726] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:51:01.117374 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:01.117043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:49543)
0925 00:51:01.117247 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:01.117292 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20260925 00:51:01.323408 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:01.322808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:34933)
0925 00:51:01.323206 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:01.323274 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":79,"threads_started":1}
I20260925 00:51:01.957415 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:01.956964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:59159)
0925 00:51:01.957202 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:01.957255 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260925 00:51:02.118413 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:02.118038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:58459)
0925 00:51:02.118261 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:02.118318 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260925 00:51:02.324522 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:02.324106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:39763)
0925 00:51:02.324336 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:02.324403 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260925 00:51:02.958626 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:02.958217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:44443)
0925 00:51:02.958465 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:02.958518 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260925 00:51:03.119547 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:03.119143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:44177)
0925 00:51:03.119401 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:03.119455 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260925 00:51:03.325789 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:03.325276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:58301)
0925 00:51:03.325615 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:03.325684 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":115,"threads_started":1}
I20260925 00:51:03.335040 16327 raft_consensus.cc:482] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:51:03.335158 16327 raft_consensus.cc:504] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:51:03.335596 16327 leader_election.cc:288] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289), 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617)
W20260925 00:51:03.336349 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
W20260925 00:51:03.336522 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
I20260925 00:51:03.336586 15933 leader_election.cc:302] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 604494106bcd4573aabbfb1b93b656e2; no voters: 8f80559ea0ce4b1ba9f4a996726e7392, 9fda2bc1e81248d7ba08065a7190b50f
I20260925 00:51:03.336748 16327 raft_consensus.cc:2726] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:51:03.959789 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:03.959396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:39669)
0925 00:51:03.959635 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:03.959684 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260925 00:51:04.120612 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:04.120225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:36561)
0925 00:51:04.120457 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:04.120510 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260925 00:51:04.327049 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:04.326584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:55463)
0925 00:51:04.326868 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:04.326953 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":106,"threads_started":1}
I20260925 00:51:04.961012 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:04.960587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:54317)
0925 00:51:04.960861 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:04.960919 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":68,"threads_started":1}
I20260925 00:51:05.121903 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:05.121473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:52135)
0925 00:51:05.121748 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:05.121810 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":102,"threads_started":1}
I20260925 00:51:05.328248 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:05.327758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:46031)
0925 00:51:05.328047 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:05.328117 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260925 00:51:05.962198 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:05.961799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:43969)
0925 00:51:05.962046 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:05.962103 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260925 00:51:06.122977 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:06.122557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:46437)
0925 00:51:06.122812 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:06.122881 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
I20260925 00:51:06.243217 16336 raft_consensus.cc:482] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:51:06.243355 16336 raft_consensus.cc:504] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:51:06.243830 16336 leader_election.cc:288] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289), 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617)
W20260925 00:51:06.244686 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
W20260925 00:51:06.244844 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
I20260925 00:51:06.244906 15933 leader_election.cc:302] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 604494106bcd4573aabbfb1b93b656e2; no voters: 8f80559ea0ce4b1ba9f4a996726e7392, 9fda2bc1e81248d7ba08065a7190b50f
I20260925 00:51:06.245083 16336 raft_consensus.cc:2726] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:51:06.329638 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:06.329032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:47679)
0925 00:51:06.329475 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:06.329541 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":88,"threads_started":1}
I20260925 00:51:06.963630 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:06.963139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:38801)
0925 00:51:06.963468 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:06.963534 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":73,"threads_started":1}
I20260925 00:51:07.124401 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:07.123879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:33323)
0925 00:51:07.124217 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:07.124284 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":89,"threads_started":1}
I20260925 00:51:07.331162 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:07.330647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:48741)
0925 00:51:07.331006 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:07.331067 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":88,"threads_started":1}
I20260925 00:51:07.965161 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:07.964693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:35259)
0925 00:51:07.964998 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:07.965066 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":80,"threads_started":1}
I20260925 00:51:08.125739 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:08.125276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:44159)
0925 00:51:08.125578 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:08.125643 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":153,"threads_started":1}
I20260925 00:51:08.332581 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:08.332140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:46263)
0925 00:51:08.332422 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:08.332484 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
I20260925 00:51:08.966439 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:08.966028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:51645)
0925 00:51:08.966288 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:08.966344 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":99,"threads_started":1}
I20260925 00:51:09.126880 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:09.126481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:55517)
0925 00:51:09.126722 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:09.126775 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":84,"threads_started":1}
I20260925 00:51:09.333833 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:09.333364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:46521)
0925 00:51:09.333680 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:09.333735 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260925 00:51:09.967705 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:09.967229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:51193)
0925 00:51:09.967551 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:09.967610 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":86,"threads_started":1}
I20260925 00:51:10.059298 16348 raft_consensus.cc:482] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:51:10.059458 16348 raft_consensus.cc:504] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:51:10.059962 16348 leader_election.cc:288] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289), 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617)
W20260925 00:51:10.060761 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
W20260925 00:51:10.060920 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
I20260925 00:51:10.060985 15933 leader_election.cc:302] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 604494106bcd4573aabbfb1b93b656e2; no voters: 8f80559ea0ce4b1ba9f4a996726e7392, 9fda2bc1e81248d7ba08065a7190b50f
I20260925 00:51:10.061174 16348 raft_consensus.cc:2726] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:51:10.128031 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:10.127633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:46801)
0925 00:51:10.127880 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:10.127936 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260925 00:51:10.335016 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:10.334574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:50563)
0925 00:51:10.334865 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:10.334922 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260925 00:51:10.968945 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:10.968481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:45253)
0925 00:51:10.968742 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:10.968796 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
I20260925 00:51:11.129168 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:11.128771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:33473)
0925 00:51:11.129022 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:11.129077 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":67,"threads_started":1}
I20260925 00:51:11.336153 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:11.335727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:53371)
0925 00:51:11.336002 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:11.336058 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
I20260925 00:51:11.970130 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:11.969743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:56541)
0925 00:51:11.969986 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:11.970039 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260925 00:51:12.130262 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:12.129897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:53105)
0925 00:51:12.130120 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:12.130172 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260925 00:51:12.337303 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:12.336910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:51571)
0925 00:51:12.337144 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:12.337204 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260925 00:51:12.477569 16357 raft_consensus.cc:482] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:51:12.477721 16357 raft_consensus.cc:504] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda2bc1e81248d7ba08065a7190b50f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42289 } } peers { permanent_uuid: "8f80559ea0ce4b1ba9f4a996726e7392" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 40617 } } peers { permanent_uuid: "604494106bcd4573aabbfb1b93b656e2" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35061 } }
I20260925 00:51:12.478114 16357 leader_election.cc:288] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289), 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617)
W20260925 00:51:12.478873 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9fda2bc1e81248d7ba08065a7190b50f (127.0.249.67:42289): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
W20260925 00:51:12.479013 15933 leader_election.cc:341] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f80559ea0ce4b1ba9f4a996726e7392 (127.0.249.66:40617): Not found: Tablet not found: e4d98164a8124253aca2eeeadc901bd5
I20260925 00:51:12.479072 15933 leader_election.cc:302] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 604494106bcd4573aabbfb1b93b656e2; no voters: 8f80559ea0ce4b1ba9f4a996726e7392, 9fda2bc1e81248d7ba08065a7190b50f
I20260925 00:51:12.479219 16357 raft_consensus.cc:2726] T e4d98164a8124253aca2eeeadc901bd5 P 604494106bcd4573aabbfb1b93b656e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:51:12.971359 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:12.970845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:43305)
0925 00:51:12.971186 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:12.971243 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":67,"threads_started":1}
I20260925 00:51:13.131306 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:13.130929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:46323)
0925 00:51:13.131163 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:13.131216 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260925 00:51:13.338423 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:13.338050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:51521)
0925 00:51:13.338274 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:13.338327 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260925 00:51:13.972714 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:13.972237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:44299)
0925 00:51:13.972555 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:13.972611 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
I20260925 00:51:14.132392 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:14.131996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:55265)
0925 00:51:14.132233 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:14.132297 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260925 00:51:14.339484 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:14.339094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:42317)
0925 00:51:14.339337 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:14.339392 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260925 00:51:14.974148 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:14.973780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.65:49483)
0925 00:51:14.974008 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:14.974059 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260925 00:51:15.133687 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:15.133215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.66:60805)
0925 00:51:15.133538 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:15.133594 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":83,"threads_started":1}
I20260925 00:51:15.329331 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 15880
I20260925 00:51:15.335959 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 16047
I20260925 00:51:15.340855 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:15.340398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36105 (local address 127.0.249.67:47735)
0925 00:51:15.340698 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:15.340760 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260925 00:51:15.341836 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 16181
2026-09-25T00:51:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19290 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:51:15 dist-test-slave-05m0 krb5kdc[16369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:51:15 dist-test-slave-05m0 krb5kdc[16369](info): set up 2 sockets
Sep 25 00:51:15 dist-test-slave-05m0 krb5kdc[16369](info): commencing operation
krb5kdc: starting...
W20260925 00:51:16.409683 997 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:51:16 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:51:16.528620 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39319
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:51:16.643508 16382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:16.643854 16382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:16.643903 16382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:16.648391 16382 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 00:51:16.648448 16382 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 00:51:16.648468 16382 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 00:51:16.648484 16382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:51:16.648504 16382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:16.648535 16382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:16.648561 16382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:51:16.648586 16382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:51:16.653832 16382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:39319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:16.654742 16382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:16.655687 16382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:16.663874 16387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:16.663884 16388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:16.663883 16390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:16.664178 16382 server_base.cc:1034] running on GCE node
I20260925 00:51:16.664700 16382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:16.665843 16382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:16.667024 16382 hybrid_clock.cc:648] HybridClock initialized: now 1790297476667013 us; error 24 us; skew 500 ppm
Sep 25 00:51:16 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:16.669612 16382 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:51:16.670418 16382 webserver.cc:460] Webserver started at http://127.0.249.126:43739/ using document root <none> and password file <none>
I20260925 00:51:16.670971 16382 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:16.671060 16382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:16.671285 16382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:51:16.673327 16382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "94fd4814158f4098892f59619919fa63"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "65367f80b3d2bdf200bd0f3ec5c7442e"
server_key_iv: "5af91dc9d1f6c79a36fd32e922540e46"
server_key_version: "encryptionkey@0"
I20260925 00:51:16.673916 16382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "94fd4814158f4098892f59619919fa63"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "65367f80b3d2bdf200bd0f3ec5c7442e"
server_key_iv: "5af91dc9d1f6c79a36fd32e922540e46"
server_key_version: "encryptionkey@0"
I20260925 00:51:16.677310 16382 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:51:16.679602 16397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:16.680759 16382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:51:16.680902 16382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "94fd4814158f4098892f59619919fa63"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "65367f80b3d2bdf200bd0f3ec5c7442e"
server_key_iv: "5af91dc9d1f6c79a36fd32e922540e46"
server_key_version: "encryptionkey@0"
I20260925 00:51:16.681007 16382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:16.712126 16382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:16.714959 16382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:16.715158 16382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:16.721935 16382 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:39319
I20260925 00:51:16.722220 16449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:39319 every 8 connection(s)
I20260925 00:51:16.722878 16382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:51:16.724684 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 16382
I20260925 00:51:16.724797 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:51:16.725095 997 external_mini_cluster.cc:1428] Setting key 4f1c55aa99f897d82a972514efed6e04
I20260925 00:51:16.725924 16450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:51:16.731724 16450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Bootstrap starting.
Sep 25 00:51:16 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:51:16.734089 16450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Neither blocks nor log segments found. Creating new log.
I20260925 00:51:16.734977 16450 log.cc:824] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:16.737123 16450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: No bootstrap required, opened a new log
I20260925 00:51:16.739096 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:16.726429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.0.249.126:39319)
0925 00:51:16.726832 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:16.726852 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:16.727074 (+ 222us) server_negotiation.cc:407] Connection header received
0925 00:51:16.727850 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:16.727866 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:16.728175 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:16.728508 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:16.729289 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:16.730349 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:16.731204 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:16.731676 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:16.731844 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:16.734580 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:16.734585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:16.734598 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:16.734624 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:16.736341 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:16.736818 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:16.736823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:16.736827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:16.736926 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:16.737259 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:16.737262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:16.737263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:16.737682 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:16.737832 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:16.738122 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 00:51:16.738409 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
I20260925 00:51:16.741075 16450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:16.741317 16450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:51:16.741420 16450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fd4814158f4098892f59619919fa63, State: Initialized, Role: FOLLOWER
I20260925 00:51:16.741869 16450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:16.742019 16450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:51:16.742105 16450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:51:16.742201 16450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:51:16.743059 16450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:16.743409 16450 leader_election.cc:302] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94fd4814158f4098892f59619919fa63; no voters:
I20260925 00:51:16.743702 16450 leader_election.cc:288] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:51:16.743824 16455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:51:16.744076 16455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 1 LEADER]: Becoming Leader. State: Replica: 94fd4814158f4098892f59619919fa63, State: Running, Role: LEADER
I20260925 00:51:16.744427 16455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:16.745023 16450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:51:16.746258 16457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94fd4814158f4098892f59619919fa63. Latest consensus state: current_term: 1 leader_uuid: "94fd4814158f4098892f59619919fa63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } } }
I20260925 00:51:16.746387 16457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: This master's current role is: LEADER
I20260925 00:51:16.746610 16456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94fd4814158f4098892f59619919fa63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } } }
I20260925 00:51:16.746757 16456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: This master's current role is: LEADER
I20260925 00:51:16.746785 16464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:51:16.749624 16464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:51:16.754617 16464 catalog_manager.cc:1347] Generated new cluster ID: f56ec958673149759dbd815dc051b1ad
I20260925 00:51:16.754681 16464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:51:16.761416 16464 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:51:16.762352 16464 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:51:16.767386 16464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Generated new TSK 0
I20260925 00:51:16.768031 16464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:51:16.800045 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39319
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:51:16.911059 16476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:16.911302 16476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:16.911343 16476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:16.915357 16476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:16.915423 16476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:16.915508 16476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:51:16.920535 16476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:16.921517 16476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:16.922526 16476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:16.931805 16484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:16.931843 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:16.931964 16476 server_base.cc:1034] running on GCE node
W20260925 00:51:16.931821 16481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:16.932475 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:16.933068 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:16.934219 16476 hybrid_clock.cc:648] HybridClock initialized: now 1790297476934204 us; error 24 us; skew 500 ppm
Sep 25 00:51:16 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:16.936764 16476 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:51:16.937676 16476 webserver.cc:460] Webserver started at http://127.0.249.65:42481/ using document root <none> and password file <none>
I20260925 00:51:16.938201 16476 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:16.938288 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:16.938510 16476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:51:16.940436 16476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "4f23e7ef362c46e08c6708bbf8afd2c8"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "1289cae27b711e540c4fd6ab351bf78c"
server_key_iv: "0381d72f751d0b235ea12a84981087d8"
server_key_version: "encryptionkey@0"
I20260925 00:51:16.940987 16476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "4f23e7ef362c46e08c6708bbf8afd2c8"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "1289cae27b711e540c4fd6ab351bf78c"
server_key_iv: "0381d72f751d0b235ea12a84981087d8"
server_key_version: "encryptionkey@0"
I20260925 00:51:16.945329 16476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 00:51:16.947816 16491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:16.949004 16476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:51:16.949133 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "4f23e7ef362c46e08c6708bbf8afd2c8"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "1289cae27b711e540c4fd6ab351bf78c"
server_key_iv: "0381d72f751d0b235ea12a84981087d8"
server_key_version: "encryptionkey@0"
I20260925 00:51:16.949241 16476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:16.973968 16476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:16.977013 16476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:16.977216 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:16.977823 16476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:16.978614 16476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:51:16.978684 16476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:16.978753 16476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:51:16.978811 16476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:16.987624 16476 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:32805
I20260925 00:51:16.987645 16604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:32805 every 8 connection(s)
I20260925 00:51:16.988507 16476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:51:16.996201 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 16476
I20260925 00:51:16.996351 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:51:16.997017 997 external_mini_cluster.cc:1428] Setting key 38a3e0c8515b347e2665fc811f31dda6
Sep 25 00:51:16 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:51:17.002224 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:16.990256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:33913 (local address 127.0.249.126:39319)
0925 00:51:16.990415 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:16.990418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:16.991206 (+ 788us) server_negotiation.cc:407] Connection header received
0925 00:51:16.992509 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:16.992512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:16.992564 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:16.992639 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:16.993833 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:16.994370 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:16.995445 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:16.995800 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:16.995880 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:16.998644 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:16.998647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:16.998649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:16.998670 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:17.000104 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:17.000608 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:17.000613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:17.000615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:17.000702 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:17.001046 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:17.001050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:17.001052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:17.001234 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:17.001362 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.001995 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.002106 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 00:51:17.003448 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:16.990618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:33913)
0925 00:51:16.991036 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:16.991076 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:16.992304 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:16.992813 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:16.992819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:16.993167 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:16.993700 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:16.993713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:16.994472 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:16.994474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:16.995345 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:16.995354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:16.995943 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:16.995949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:16.996171 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:16.996847 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:16.996870 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:16.998473 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:17.000218 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:17.000224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:17.000238 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:17.000503 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:17.000808 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:17.000811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:17.000813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:17.000951 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:17.001358 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:17.001365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:17.001716 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 00:51:17.001943 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 00:51:17.002299 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260925 00:51:17.004776 16605 heartbeater.cc:347] Connected to a master server at 127.0.249.126:39319
I20260925 00:51:17.005071 16605 heartbeater.cc:464] Registering TS with master...
I20260925 00:51:17.005702 16605 heartbeater.cc:511] Master 127.0.249.126:39319 requested a full tablet report, sending...
I20260925 00:51:17.007472 16414 ts_manager.cc:194] Registered new tserver with Master: 4f23e7ef362c46e08c6708bbf8afd2c8 (127.0.249.65:32805)
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:51:17.008715 16414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:33913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:51:17.025043 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39319
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:51:17.135637 16610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:17.135888 16610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:17.135931 16610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:17.140040 16610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:17.140100 16610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:17.140157 16610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:51:17.145375 16610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:17.146438 16610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:17.147411 16610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:51:17.156973 16610 server_base.cc:1034] running on GCE node
W20260925 00:51:17.156939 16615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:17.156965 16618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:17.156939 16616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:17.157611 16610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:17.158210 16610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:17.159368 16610 hybrid_clock.cc:648] HybridClock initialized: now 1790297477159318 us; error 69 us; skew 500 ppm
Sep 25 00:51:17 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:17.162029 16610 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:51:17.162894 16610 webserver.cc:460] Webserver started at http://127.0.249.66:37945/ using document root <none> and password file <none>
I20260925 00:51:17.163429 16610 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:17.163515 16610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:17.163735 16610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:51:17.165838 16610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "4f71c26c19624a9ca679a0b95778638c"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "f4e6220cd030abb7b9c9aad04e0ce58f"
server_key_iv: "b88edebb4a808492f05453c56b8435cf"
server_key_version: "encryptionkey@0"
I20260925 00:51:17.166371 16610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "4f71c26c19624a9ca679a0b95778638c"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "f4e6220cd030abb7b9c9aad04e0ce58f"
server_key_iv: "b88edebb4a808492f05453c56b8435cf"
server_key_version: "encryptionkey@0"
I20260925 00:51:17.169922 16610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:51:17.172186 16625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:17.173373 16610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:51:17.173581 16610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "4f71c26c19624a9ca679a0b95778638c"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "f4e6220cd030abb7b9c9aad04e0ce58f"
server_key_iv: "b88edebb4a808492f05453c56b8435cf"
server_key_version: "encryptionkey@0"
I20260925 00:51:17.173719 16610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:17.205044 16610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:17.208168 16610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:17.208372 16610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:17.208964 16610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:17.209789 16610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:51:17.209858 16610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:17.209933 16610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:51:17.209985 16610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:17.218683 16610 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:43413
I20260925 00:51:17.218708 16738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:43413 every 8 connection(s)
I20260925 00:51:17.219530 16610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:51:17.221202 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 16610
I20260925 00:51:17.221349 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:51:17.221638 997 external_mini_cluster.cc:1428] Setting key decc0826fa1a819d93e380fa6426cfa5
Sep 25 00:51:17 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:51:17.233268 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.221238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:59537 (local address 127.0.249.126:39319)
0925 00:51:17.221405 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:17.221410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:17.222307 (+ 897us) server_negotiation.cc:407] Connection header received
0925 00:51:17.223661 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:17.223667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:17.223728 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:17.223830 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:17.224933 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.225518 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.226636 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.226878 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.226986 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.229770 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:17.229773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:17.229783 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:17.229805 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:17.231294 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:17.231741 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:17.231743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:17.231744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:17.231848 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:17.232188 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:17.232193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:17.232195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:17.232369 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:17.232473 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.232986 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.233106 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
I20260925 00:51:17.234205 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.221655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:59537)
0925 00:51:17.222147 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:17.222200 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:17.223455 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:17.223969 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:17.223975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:17.224338 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:17.224792 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.224806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.225636 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.225640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.226511 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.226519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.227022 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.227026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.227196 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.227822 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:17.227837 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:17.229604 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:17.231400 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:17.231404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:17.231416 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:17.231655 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:17.231945 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:17.231949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:17.231951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:17.232091 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:17.232471 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:17.232477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:17.232718 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 00:51:17.232939 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 00:51:17.233305 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":140,"threads_started":1}
I20260925 00:51:17.235435 16739 heartbeater.cc:347] Connected to a master server at 127.0.249.126:39319
I20260925 00:51:17.235694 16739 heartbeater.cc:464] Registering TS with master...
I20260925 00:51:17.236183 16739 heartbeater.cc:511] Master 127.0.249.126:39319 requested a full tablet report, sending...
I20260925 00:51:17.237231 16414 ts_manager.cc:194] Registered new tserver with Master: 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413)
I20260925 00:51:17.237841 16414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:59537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:51:17.252641 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39319
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:51:17.367686 16744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:17.367971 16744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:17.368045 16744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:17.371953 16744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:17.372038 16744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:17.372128 16744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:51:17.377269 16744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:17.378304 16744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:17.379292 16744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:17.388797 16752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:17.388839 16749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:17.388880 16750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:17.389127 16744 server_base.cc:1034] running on GCE node
I20260925 00:51:17.389616 16744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:17.390214 16744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:17.391404 16744 hybrid_clock.cc:648] HybridClock initialized: now 1790297477391392 us; error 30 us; skew 500 ppm
Sep 25 00:51:17 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:17.394025 16744 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:51:17.394827 16744 webserver.cc:460] Webserver started at http://127.0.249.67:46169/ using document root <none> and password file <none>
I20260925 00:51:17.395365 16744 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:17.395452 16744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:17.395673 16744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:51:17.397786 16744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "242a18419445472dacd325f2a983bcea"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "7b1ac761732966b4d9b9d00a7be1bfa3"
server_key_iv: "b42933354e5e0fb47759af99e5d46971"
server_key_version: "encryptionkey@0"
I20260925 00:51:17.398315 16744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "242a18419445472dacd325f2a983bcea"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "7b1ac761732966b4d9b9d00a7be1bfa3"
server_key_iv: "b42933354e5e0fb47759af99e5d46971"
server_key_version: "encryptionkey@0"
I20260925 00:51:17.401896 16744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:51:17.404156 16759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:17.405306 16744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:51:17.405476 16744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "242a18419445472dacd325f2a983bcea"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "7b1ac761732966b4d9b9d00a7be1bfa3"
server_key_iv: "b42933354e5e0fb47759af99e5d46971"
server_key_version: "encryptionkey@0"
I20260925 00:51:17.405592 16744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:17.426949 16744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:17.429916 16744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:17.430159 16744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:17.430737 16744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:17.431535 16744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:51:17.431613 16744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:17.431702 16744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:51:17.431754 16744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:17.440042 16744 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:33331
I20260925 00:51:17.440068 16872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:33331 every 8 connection(s)
I20260925 00:51:17.440878 16744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:51:17.449579 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 16744
I20260925 00:51:17.449694 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:51:17.449955 997 external_mini_cluster.cc:1428] Setting key 5130ed4b59034c9ef393fa2051cb9589
Sep 25 00:51:17 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:51:17.454306 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.442598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:45837 (local address 127.0.249.126:39319)
0925 00:51:17.442739 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:17.442744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:17.443584 (+ 840us) server_negotiation.cc:407] Connection header received
0925 00:51:17.444861 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:17.444864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:17.444907 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:17.444978 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:17.446179 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.446730 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.447871 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.448081 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.448158 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.450685 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:17.450689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:17.450691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:17.450719 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:17.452418 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:17.452854 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:17.452856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:17.452857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:17.452925 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:17.453236 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:17.453239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:17.453240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:17.453458 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:17.453554 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.454027 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.454144 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:51:17.455044 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.443018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:45837)
0925 00:51:17.443437 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:17.443479 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:17.444665 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:17.445124 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:17.445130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:17.445552 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:17.446053 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.446066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.446851 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.446853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.447764 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.447773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.448199 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.448202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.448352 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.448930 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:17.448945 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:17.450526 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:17.452517 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:17.452521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:17.452533 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:17.452769 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:17.453009 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:17.453015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:17.453018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:17.453158 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:17.453555 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:17.453562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:17.453792 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 00:51:17.453985 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 00:51:17.454297 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
I20260925 00:51:17.456049 16873 heartbeater.cc:347] Connected to a master server at 127.0.249.126:39319
I20260925 00:51:17.456286 16873 heartbeater.cc:464] Registering TS with master...
I20260925 00:51:17.456780 16873 heartbeater.cc:511] Master 127.0.249.126:39319 requested a full tablet report, sending...
I20260925 00:51:17.457777 16414 ts_manager.cc:194] Registered new tserver with Master: 242a18419445472dacd325f2a983bcea (127.0.249.67:33331)
I20260925 00:51:17.458304 16414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:45837
I20260925 00:51:17.463972 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:51:17.473188 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.466061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.0.249.126:39319)
0925 00:51:17.466195 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:17.466198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:17.466467 (+ 269us) server_negotiation.cc:407] Connection header received
0925 00:51:17.466581 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:17.466583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:17.466619 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:17.466684 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:17.467384 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.467891 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.468727 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.468944 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.469017 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.470436 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:17.470439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:17.470441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:17.470462 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:17.471702 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:17.472084 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:17.472087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:17.472088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:17.472158 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:17.472455 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:17.472457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:17.472458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:17.472663 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:17.472762 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.472987 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.473074 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:51:17.476029 16414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:51:17.479285 16414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:51:17.493141 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.487385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:32805 (local address 127.0.0.1:38706)
0925 00:51:17.488045 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:17.488099 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:17.488234 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:17.489081 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:17.489086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:17.489107 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:51:17.489227 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.489232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.490434 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.490451 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.491667 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.491679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.492543 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.492547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.492660 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.492701 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 00:51:17.492854 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 00:51:17.492979 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260925 00:51:17.493160 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.487519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.249.65:32805)
0925 00:51:17.488352 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:17.488369 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:17.488401 (+ 32us) server_negotiation.cc:407] Connection header received
0925 00:51:17.488467 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:17.488471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:17.488702 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:17.488868 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:51:17.489387 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.490291 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.491814 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.492407 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.492545 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.492709 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.492889 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.492988 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":170,"threads_started":1}
I20260925 00:51:17.494143 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.487659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:33331 (local address 127.0.0.1:57200)
0925 00:51:17.488804 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:17.488817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:17.488889 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:17.489343 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:17.489348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:17.489367 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:51:17.489607 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.489615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.491107 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.491110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.492446 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.492452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.493682 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.493687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.493792 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.493819 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:51:17.493909 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 00:51:17.493975 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":193,"thread_start_us":73,"threads_started":1}
I20260925 00:51:17.494179 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.487423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:43413 (local address 127.0.0.1:57122)
0925 00:51:17.488729 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:17.488743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:17.488853 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:17.489448 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:17.489453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:17.489467 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:51:17.489621 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.489628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.491107 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.491110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.492438 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.492449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.493750 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.493754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.493900 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.493927 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:51:17.493998 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 00:51:17.494060 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"mutex_wait_us":77,"thread_start_us":99,"threads_started":1}
I20260925 00:51:17.494189 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.488137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.0.249.67:33331)
0925 00:51:17.488587 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:17.488599 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:17.488974 (+ 375us) server_negotiation.cc:407] Connection header received
0925 00:51:17.489037 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:17.489041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:17.489198 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:17.489343 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:51:17.489761 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.490954 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.492579 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.493567 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.493662 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.493788 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.493949 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.494046 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":89,"threads_started":1}
I20260925 00:51:17.494680 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.488137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.0.249.66:43413)
0925 00:51:17.488689 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:17.488699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:17.488948 (+ 249us) server_negotiation.cc:407] Connection header received
0925 00:51:17.489025 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:17.489030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:17.489198 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:17.489341 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:51:17.489793 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.490954 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.492579 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.493446 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.493549 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.493688 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.494431 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.494540 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":127,"threads_started":1}
I20260925 00:51:17.496488 16539 tablet_service.cc:1467] Processing CreateTablet for tablet 0a8baeb996a140a384f642e1a792db3b (DEFAULT_TABLE table=test-table [id=1552d62d9a4c4496aa6d323a794da5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:51:17.497112 16807 tablet_service.cc:1467] Processing CreateTablet for tablet 0a8baeb996a140a384f642e1a792db3b (DEFAULT_TABLE table=test-table [id=1552d62d9a4c4496aa6d323a794da5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:51:17.497731 16539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a8baeb996a140a384f642e1a792db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:51:17.498194 16807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a8baeb996a140a384f642e1a792db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:51:17.499204 16673 tablet_service.cc:1467] Processing CreateTablet for tablet 0a8baeb996a140a384f642e1a792db3b (DEFAULT_TABLE table=test-table [id=1552d62d9a4c4496aa6d323a794da5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:51:17.499981 16673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a8baeb996a140a384f642e1a792db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:51:17.505313 16890 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Bootstrap starting.
I20260925 00:51:17.506186 16891 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: Bootstrap starting.
I20260925 00:51:17.506664 16892 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Bootstrap starting.
I20260925 00:51:17.508045 16890 tablet_bootstrap.cc:654] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Neither blocks nor log segments found. Creating new log.
I20260925 00:51:17.508481 16891 tablet_bootstrap.cc:654] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: Neither blocks nor log segments found. Creating new log.
I20260925 00:51:17.508970 16892 tablet_bootstrap.cc:654] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Neither blocks nor log segments found. Creating new log.
I20260925 00:51:17.509487 16891 log.cc:824] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:17.509630 16890 log.cc:824] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:17.509745 16892 log.cc:824] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:17.511355 16890 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: No bootstrap required, opened a new log
I20260925 00:51:17.511365 16891 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: No bootstrap required, opened a new log
I20260925 00:51:17.511550 16890 ts_tablet_manager.cc:1397] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:51:17.511561 16891 ts_tablet_manager.cc:1397] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:51:17.514840 16892 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: No bootstrap required, opened a new log
I20260925 00:51:17.514684 16891 raft_consensus.cc:348] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:17.514940 16891 raft_consensus.cc:374] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:51:17.515002 16891 raft_consensus.cc:729] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242a18419445472dacd325f2a983bcea, State: Initialized, Role: FOLLOWER
I20260925 00:51:17.515021 16892 ts_tablet_manager.cc:1397] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 00:51:17.515161 16890 raft_consensus.cc:348] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:17.515420 16890 raft_consensus.cc:374] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:51:17.515470 16890 raft_consensus.cc:729] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f23e7ef362c46e08c6708bbf8afd2c8, State: Initialized, Role: FOLLOWER
I20260925 00:51:17.515489 16891 consensus_queue.cc:260] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:17.515897 16890 consensus_queue.cc:260] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:17.516258 16873 heartbeater.cc:503] Master 127.0.249.126:39319 was elected leader, sending a full tablet report...
I20260925 00:51:17.516552 16891 ts_tablet_manager.cc:1428] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:51:17.516810 16890 ts_tablet_manager.cc:1428] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:51:17.517163 16605 heartbeater.cc:503] Master 127.0.249.126:39319 was elected leader, sending a full tablet report...
I20260925 00:51:17.519579 16892 raft_consensus.cc:348] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:17.519855 16892 raft_consensus.cc:374] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:51:17.519924 16892 raft_consensus.cc:729] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f71c26c19624a9ca679a0b95778638c, State: Initialized, Role: FOLLOWER
I20260925 00:51:17.520331 16892 consensus_queue.cc:260] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:17.521260 16739 heartbeater.cc:503] Master 127.0.249.126:39319 was elected leader, sending a full tablet report...
I20260925 00:51:17.521317 16892 ts_tablet_manager.cc:1428] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:51:17.651943 16898 raft_consensus.cc:482] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:51:17.652181 16898 raft_consensus.cc:504] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:17.653375 16898 leader_election.cc:288] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f23e7ef362c46e08c6708bbf8afd2c8 (127.0.249.65:32805), 242a18419445472dacd325f2a983bcea (127.0.249.67:33331)
I20260925 00:51:17.657001 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.653655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:33331 (local address 127.0.249.66:55527)
0925 00:51:17.653767 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:17.653799 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:17.653896 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:17.654195 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:17.654198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:17.654224 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:51:17.654347 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.654355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.655088 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.655093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.655917 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.655926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.656567 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.656571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.656672 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.656698 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:51:17.656802 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 00:51:17.656872 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":33}
I20260925 00:51:17.657001 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.653787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:55527 (local address 127.0.249.67:33331)
0925 00:51:17.653911 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:17.653916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:17.653932 (+ 16us) server_negotiation.cc:407] Connection header received
0925 00:51:17.654003 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:17.654007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:17.654062 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:17.654141 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:51:17.654499 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.654973 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.656041 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.656450 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.656576 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.656631 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.656819 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.656882 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:51:17.657532 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.653656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:32805 (local address 127.0.249.66:33401)
0925 00:51:17.654345 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:17.654360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:17.654462 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:17.654733 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:17.654736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:17.654753 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:51:17.654861 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.654865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.655579 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.655582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.656385 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:17.656394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.657050 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.657054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:17.657158 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.657186 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 00:51:17.657289 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 00:51:17.657370 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":330,"thread_start_us":125,"threads_started":1}
I20260925 00:51:17.657536 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.653757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:33401 (local address 127.0.249.65:32805)
0925 00:51:17.653900 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:17.653904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:17.654444 (+ 540us) server_negotiation.cc:407] Connection header received
0925 00:51:17.654583 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:17.654592 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:17.654637 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:17.654722 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:51:17.654969 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.655461 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.656508 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.656915 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.657050 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.657097 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.657332 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.657448 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:51:17.657701 16827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a8baeb996a140a384f642e1a792db3b" candidate_uuid: "4f71c26c19624a9ca679a0b95778638c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "242a18419445472dacd325f2a983bcea" is_pre_election: true
I20260925 00:51:17.658061 16827 raft_consensus.cc:2454] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f71c26c19624a9ca679a0b95778638c in term 0.
I20260925 00:51:17.658233 16559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a8baeb996a140a384f642e1a792db3b" candidate_uuid: "4f71c26c19624a9ca679a0b95778638c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" is_pre_election: true
I20260925 00:51:17.658516 16559 raft_consensus.cc:2454] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f71c26c19624a9ca679a0b95778638c in term 0.
I20260925 00:51:17.658548 16629 leader_election.cc:302] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242a18419445472dacd325f2a983bcea, 4f71c26c19624a9ca679a0b95778638c; no voters:
I20260925 00:51:17.658874 16898 raft_consensus.cc:2781] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:51:17.658972 16898 raft_consensus.cc:482] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:51:17.659016 16898 raft_consensus.cc:3037] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:51:17.660017 16898 raft_consensus.cc:504] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:17.660344 16898 leader_election.cc:288] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [CANDIDATE]: Term 1 election: Requested vote from peers 4f23e7ef362c46e08c6708bbf8afd2c8 (127.0.249.65:32805), 242a18419445472dacd325f2a983bcea (127.0.249.67:33331)
I20260925 00:51:17.660620 16559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a8baeb996a140a384f642e1a792db3b" candidate_uuid: "4f71c26c19624a9ca679a0b95778638c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8"
I20260925 00:51:17.660729 16559 raft_consensus.cc:3037] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:51:17.660769 16827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a8baeb996a140a384f642e1a792db3b" candidate_uuid: "4f71c26c19624a9ca679a0b95778638c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "242a18419445472dacd325f2a983bcea"
I20260925 00:51:17.660924 16827 raft_consensus.cc:3037] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:51:17.661931 16559 raft_consensus.cc:2454] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f71c26c19624a9ca679a0b95778638c in term 1.
I20260925 00:51:17.662070 16827 raft_consensus.cc:2454] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f71c26c19624a9ca679a0b95778638c in term 1.
I20260925 00:51:17.662287 16626 leader_election.cc:302] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f23e7ef362c46e08c6708bbf8afd2c8, 4f71c26c19624a9ca679a0b95778638c; no voters:
I20260925 00:51:17.662485 16898 raft_consensus.cc:2781] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:51:17.662688 16898 raft_consensus.cc:686] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 1 LEADER]: Becoming Leader. State: Replica: 4f71c26c19624a9ca679a0b95778638c, State: Running, Role: LEADER
I20260925 00:51:17.662995 16898 consensus_queue.cc:237] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:17.666165 16413 catalog_manager.cc:5495] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f71c26c19624a9ca679a0b95778638c (127.0.249.66). New cstate: current_term: 1 leader_uuid: "4f71c26c19624a9ca679a0b95778638c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } health_report { overall_health: UNKNOWN } } }
I20260925 00:51:17.680760 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.677764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.0.249.66:43413)
0925 00:51:17.677921 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:17.677926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:17.677937 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:51:17.678015 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:17.678017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:17.678066 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:17.678141 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:51:17.678342 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.678815 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.679598 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:17.679794 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:17.679865 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:17.680095 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:51:17.680478 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:51:17.680562 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:17.680621 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 00:51:17.680664 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 00:51:17.686360 16827 raft_consensus.cc:1261] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [term 1 FOLLOWER]: Refusing update from remote peer 4f71c26c19624a9ca679a0b95778638c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:51:17.686374 16559 raft_consensus.cc:1261] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Refusing update from remote peer 4f71c26c19624a9ca679a0b95778638c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:51:17.687027 16898 consensus_queue.cc:1035] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:51:17.687264 16900 consensus_queue.cc:1035] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [LEADER]: Connected to new peer: Peer: permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 00:51:17.691715 16874 tablet.cc:2369] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:51:17.695394 16903 mvcc.cc:205] Tried to move back new op lower bound from 7333058468595970048 to 7333058468510277632. Current Snapshot: MvccSnapshot[applied={T|T < 7333058468595970048 or (T in {7333058468595970048})}]
I20260925 00:51:17.698472 16904 mvcc.cc:205] Tried to move back new op lower bound from 7333058468595970048 to 7333058468510277632. Current Snapshot: MvccSnapshot[applied={T|T < 7333058468595970048}]
I20260925 00:51:17.700984 16414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:51:17.701148 16414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:51:17.703531 16414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Sending DeleteTablet for 3 replicas of tablet 0a8baeb996a140a384f642e1a792db3b
I20260925 00:51:17.704265 16807 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a8baeb996a140a384f642e1a792db3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:51:17 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:57200
I20260925 00:51:17.704243 16673 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a8baeb996a140a384f642e1a792db3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:51:17 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:57122
I20260925 00:51:17.704249 16539 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a8baeb996a140a384f642e1a792db3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:51:17 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:38706
I20260925 00:51:17.704650 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 16476
I20260925 00:51:17.706018 16913 tablet_replica.cc:330] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: stopping tablet replica
I20260925 00:51:17.706303 16913 raft_consensus.cc:2229] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:51:17.706586 16913 raft_consensus.cc:2258] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:51:17.707937 16913 ts_tablet_manager.cc:1905] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:51:17.711344 16913 ts_tablet_manager.cc:1918] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:51:17.711448 16913 log.cc:1196] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/0a8baeb996a140a384f642e1a792db3b
I20260925 00:51:17.711711 16915 tablet_replica.cc:330] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: stopping tablet replica
I20260925 00:51:17.711797 16913 ts_tablet_manager.cc:1939] T 0a8baeb996a140a384f642e1a792db3b P 242a18419445472dacd325f2a983bcea: Deleting consensus metadata
I20260925 00:51:17.712199 16915 raft_consensus.cc:2229] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:51:17.712599 16915 raft_consensus.cc:2258] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 00:51:17.712653 16398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:32805 (error 108)
I20260925 00:51:17.713032 16401 catalog_manager.cc:4841] TS 242a18419445472dacd325f2a983bcea (127.0.249.67:33331): tablet 0a8baeb996a140a384f642e1a792db3b (table test-table [id=1552d62d9a4c4496aa6d323a794da5db]) successfully deleted
I20260925 00:51:17.713762 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 16610
I20260925 00:51:17.714476 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.713765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:32805 (local address 127.0.0.1:38720)
0925 00:51:17.714299 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:17.714377 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:32805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426}
W20260925 00:51:17.714653 16398 catalog_manager.cc:4584] TS 4f23e7ef362c46e08c6708bbf8afd2c8 (127.0.249.65:32805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a8baeb996a140a384f642e1a792db3b: Network error: Client connection negotiation failed: client connection to 127.0.249.65:32805: connect: Connection refused (error 111)
I20260925 00:51:17.719933 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 16744
I20260925 00:51:17.720244 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:17.719949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:43413 (local address 127.0.0.1:57142)
0925 00:51:17.720090 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:17.720154 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.66:43413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 00:51:17.725554 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 16382
I20260925 00:51:37.732144 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39319
--webserver_interface=127.0.249.126
--webserver_port=43739
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:51:37.842496 16917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:37.842761 16917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:37.842832 16917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:37.846820 16917 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 00:51:37.846887 16917 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 00:51:37.846920 16917 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 00:51:37.846954 16917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:51:37.846992 16917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:37.847033 16917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:37.847069 16917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:51:37.847105 16917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:51:37.852144 16917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:39319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:37.853120 16917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:37.854081 16917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:37.866314 16924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:37.866299 16926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:37.866292 16923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:37.866814 16917 server_base.cc:1034] running on GCE node
I20260925 00:51:37.867409 16917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:37.868518 16917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:37.869697 16917 hybrid_clock.cc:648] HybridClock initialized: now 1790297497869679 us; error 28 us; skew 500 ppm
Sep 25 00:51:37 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:37.872349 16917 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:51:37.873240 16917 webserver.cc:460] Webserver started at http://127.0.249.126:43739/ using document root <none> and password file <none>
I20260925 00:51:37.873838 16917 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:37.873944 16917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:37.877743 16917 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 00:51:37.879822 16933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:37.881079 16917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:51:37.881243 16917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "94fd4814158f4098892f59619919fa63"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "65367f80b3d2bdf200bd0f3ec5c7442e"
server_key_iv: "5af91dc9d1f6c79a36fd32e922540e46"
server_key_version: "encryptionkey@0"
I20260925 00:51:37.881906 16917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:37.915526 16917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:37.918857 16917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:37.919070 16917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:37.926339 16917 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:39319
I20260925 00:51:37.926683 16985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:39319 every 8 connection(s)
I20260925 00:51:37.927456 16917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:51:37.929076 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 16917
I20260925 00:51:37.929641 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:32805
--local_ip_for_outbound_sockets=127.0.249.65
--tserver_master_addrs=127.0.249.126:39319
--webserver_port=42481
--webserver_interface=127.0.249.65
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:51:37.932937 16986 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:51:37.934695 16986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Bootstrap starting.
I20260925 00:51:37.951350 16986 log.cc:824] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:37.959838 16986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:51:37.960222 16986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Bootstrap complete.
I20260925 00:51:37.964464 16986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:37.964877 16986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fd4814158f4098892f59619919fa63, State: Initialized, Role: FOLLOWER
I20260925 00:51:37.965468 16986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:37.965628 16986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:51:37.965706 16986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:51:37.965804 16986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 1 FOLLOWER]: Advancing to term 2
I20260925 00:51:37.967679 16986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:37.968014 16986 leader_election.cc:302] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94fd4814158f4098892f59619919fa63; no voters:
I20260925 00:51:37.968343 16986 leader_election.cc:288] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 00:51:37.968469 16990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 00:51:37.968722 16990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 2 LEADER]: Becoming Leader. State: Replica: 94fd4814158f4098892f59619919fa63, State: Running, Role: LEADER
I20260925 00:51:37.969067 16990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:37.969523 16986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:51:37.970045 16990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94fd4814158f4098892f59619919fa63. Latest consensus state: current_term: 2 leader_uuid: "94fd4814158f4098892f59619919fa63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } } }
I20260925 00:51:37.970163 16990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: This master's current role is: LEADER
I20260925 00:51:37.970785 16991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "94fd4814158f4098892f59619919fa63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } } }
I20260925 00:51:37.970917 16994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:51:37.970948 16991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: This master's current role is: LEADER
I20260925 00:51:37.975514 16994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:51:37.976118 16994 catalog_manager.cc:1259] Loaded cluster ID: f56ec958673149759dbd815dc051b1ad
I20260925 00:51:37.976253 16994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:51:37.978741 16994 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:51:37.979461 16994 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Loaded TSK: 0
I20260925 00:51:37.993290 16994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Generated new TSK 1
I20260925 00:51:37.993564 16994 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:51:38.047394 16988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:38.047639 16988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:38.047681 16988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:38.051826 16988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:38.051884 16988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:38.051942 16988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:51:38.057497 16988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:32805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=42481
--tserver_master_addrs=127.0.249.126:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:38.058511 16988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:38.059602 16988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:51:38.069331 16988 server_base.cc:1034] running on GCE node
W20260925 00:51:38.069296 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:38.069296 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:38.069615 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:38.070091 16988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:38.070742 16988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:38.071954 16988 hybrid_clock.cc:648] HybridClock initialized: now 1790297498071927 us; error 46 us; skew 500 ppm
Sep 25 00:51:38 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:38.075003 16988 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:51:38.075947 16988 webserver.cc:460] Webserver started at http://127.0.249.65:42481/ using document root <none> and password file <none>
I20260925 00:51:38.076488 16988 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:38.076596 16988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:38.080542 16988 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:51:38.082731 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:38.083972 16988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:51:38.084061 16988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "4f23e7ef362c46e08c6708bbf8afd2c8"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "1289cae27b711e540c4fd6ab351bf78c"
server_key_iv: "0381d72f751d0b235ea12a84981087d8"
server_key_version: "encryptionkey@0"
I20260925 00:51:38.084654 16988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:38.105562 16988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:38.108589 16988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:38.108805 16988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:38.109454 16988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:38.110496 17030 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:51:38.114313 16988 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:51:38.114369 16988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 00:51:38.114439 16988 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:51:38.116055 16988 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:51:38.116096 16988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 00:51:38.116374 17030 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Bootstrap starting.
I20260925 00:51:38.126564 16988 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:32805
I20260925 00:51:38.126678 17137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:32805 every 8 connection(s)
I20260925 00:51:38.127506 16988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:51:38.134306 17030 log.cc:824] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:38.136371 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 16988
I20260925 00:51:38.136861 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:43413
--local_ip_for_outbound_sockets=127.0.249.66
--tserver_master_addrs=127.0.249.126:39319
--webserver_port=37945
--webserver_interface=127.0.249.66
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:51:38 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:51:38.142140 17030 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:51:38.142527 17030 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Bootstrap complete.
I20260925 00:51:38.143879 17030 ts_tablet_manager.cc:1397] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.011s
I20260925 00:51:38.146124 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.129333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:47325)
0925 00:51:38.129886 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:38.129943 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:38.131166 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:38.132307 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:38.132316 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:38.132792 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:38.133425 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.133444 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.134832 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.134836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.135862 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.135872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.136883 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.136887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.137126 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.137896 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:38.137922 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:38.139847 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:38.142643 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.142649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.142660 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.143023 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.143472 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.143478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.143484 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.143656 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.144435 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:38.144440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:38.144711 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 00:51:38.144892 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 00:51:38.145197 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":3,"thread_start_us":134,"threads_started":1}
I20260925 00:51:38.146409 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.129504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:47325 (local address 127.0.249.126:39319)
0925 00:51:38.129959 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.129976 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.130047 (+ 71us) server_negotiation.cc:407] Connection header received
0925 00:51:38.131576 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.131587 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.131876 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.132140 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:38.133594 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.134672 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.135996 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.136722 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.136910 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.140053 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:38.140062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:38.140081 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:38.140120 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:38.142490 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.143171 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.143180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.143187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.143362 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.143783 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.143790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.143793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.144300 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:38.144451 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.145069 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.145417 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":108,"threads_started":1}
I20260925 00:51:38.146652 17030 raft_consensus.cc:348] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:38.147145 17030 raft_consensus.cc:729] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f23e7ef362c46e08c6708bbf8afd2c8, State: Initialized, Role: FOLLOWER
I20260925 00:51:38.147609 17030 consensus_queue.cc:260] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:38.148463 17030 ts_tablet_manager.cc:1428] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:51:38.148998 17138 heartbeater.cc:347] Connected to a master server at 127.0.249.126:39319
I20260925 00:51:38.149215 17138 heartbeater.cc:464] Registering TS with master...
I20260925 00:51:38.149705 17138 heartbeater.cc:511] Master 127.0.249.126:39319 requested a full tablet report, sending...
I20260925 00:51:38.151886 16950 ts_manager.cc:194] Registered new tserver with Master: 4f23e7ef362c46e08c6708bbf8afd2c8 (127.0.249.65:32805)
I20260925 00:51:38.155223 16950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:47325
I20260925 00:51:38.157985 17138 heartbeater.cc:503] Master 127.0.249.126:39319 was elected leader, sending a full tablet report...
Sep 25 00:51:38 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.65@KRBTEST.COM
I20260925 00:51:38.165377 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.154503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:32805 (local address 127.0.0.1:53388)
0925 00:51:38.155170 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:38.155215 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:38.155351 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:38.156617 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:38.156622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:38.156959 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:38.157052 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.157059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.158098 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.158103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.158891 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.158902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.159674 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.159678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.159766 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.160301 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:38.160313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:38.161726 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:38.163684 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.163693 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.163698 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.164001 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.164299 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.164301 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.164302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.164399 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.164853 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:38.164857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:38.164985 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 00:51:38.165106 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 00:51:38.165254 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":105,"threads_started":1}
I20260925 00:51:38.165594 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.154609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.0.249.65:32805)
0925 00:51:38.154933 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.154944 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.156096 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 00:51:38.156250 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.156258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.156455 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.156617 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:38.157175 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.157985 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.159032 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.159541 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.159657 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.161860 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:38.161869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:38.161881 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:38.161905 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:38.163574 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.164106 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.164112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.164113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.164202 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.164466 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.164469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.164470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.164753 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:38.164876 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.165207 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.165364 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":85,"threads_started":1}
I20260925 00:51:38.166751 17072 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a8baeb996a140a384f642e1a792db3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:51:17 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:53388
I20260925 00:51:38.167587 17147 tablet_replica.cc:330] stopping tablet replica
I20260925 00:51:38.167899 17147 raft_consensus.cc:2229] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:51:38.168143 17147 raft_consensus.cc:2258] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:51:38.169195 17147 ts_tablet_manager.cc:1905] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:51:38.173683 17147 ts_tablet_manager.cc:1918] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:51:38.173792 17147 log.cc:1196] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal/wals/0a8baeb996a140a384f642e1a792db3b
I20260925 00:51:38.174085 17147 ts_tablet_manager.cc:1939] T 0a8baeb996a140a384f642e1a792db3b P 4f23e7ef362c46e08c6708bbf8afd2c8: Deleting consensus metadata
I20260925 00:51:38.175946 16934 catalog_manager.cc:4841] TS 4f23e7ef362c46e08c6708bbf8afd2c8 (127.0.249.65:32805): tablet 0a8baeb996a140a384f642e1a792db3b (table test-table [id=1552d62d9a4c4496aa6d323a794da5db]) successfully deleted
W20260925 00:51:38.253625 17143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:38.253919 17143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:38.253995 17143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:38.258913 17143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:38.259002 17143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:38.259114 17143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:51:38.264683 17143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:43413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=37945
--tserver_master_addrs=127.0.249.126:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:38.265957 17143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:38.266961 17143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:38.276638 17153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:38.276611 17155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:38.276585 17152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:38.276901 17143 server_base.cc:1034] running on GCE node
I20260925 00:51:38.277312 17143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:38.277963 17143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:38.279247 17143 hybrid_clock.cc:648] HybridClock initialized: now 1790297498279230 us; error 26 us; skew 500 ppm
Sep 25 00:51:38 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:38.281935 17143 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:51:38.282768 17143 webserver.cc:460] Webserver started at http://127.0.249.66:37945/ using document root <none> and password file <none>
I20260925 00:51:38.283316 17143 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:38.283440 17143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:38.287544 17143 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:51:38.289880 17162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:38.291090 17143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:51:38.291185 17143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "4f71c26c19624a9ca679a0b95778638c"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "f4e6220cd030abb7b9c9aad04e0ce58f"
server_key_iv: "b88edebb4a808492f05453c56b8435cf"
server_key_version: "encryptionkey@0"
I20260925 00:51:38.291728 17143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:38.307263 17143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:38.310505 17143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:38.310706 17143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:38.311288 17143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:38.312283 17169 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:51:38.316339 17143 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:51:38.316396 17143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 00:51:38.316470 17143 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:51:38.318096 17143 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:51:38.318137 17143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 00:51:38.318423 17169 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Bootstrap starting.
I20260925 00:51:38.324610 17169 log.cc:824] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:38.328183 17143 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:43413
I20260925 00:51:38.328263 17277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:43413 every 8 connection(s)
I20260925 00:51:38.329007 17143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:51:38.332690 17169 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:51:38.333099 17169 tablet_bootstrap.cc:492] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Bootstrap complete.
I20260925 00:51:38.333474 17169 ts_tablet_manager.cc:1397] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Time spent bootstrapping tablet: real 0.015s user 0.015s sys 0.000s
I20260925 00:51:38.334254 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 17143
I20260925 00:51:38.334726 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:33331
--local_ip_for_outbound_sockets=127.0.249.67
--tserver_master_addrs=127.0.249.126:39319
--webserver_port=46169
--webserver_interface=127.0.249.67
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:51:38.336982 17169 raft_consensus.cc:348] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
I20260925 00:51:38.337584 17169 raft_consensus.cc:729] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f71c26c19624a9ca679a0b95778638c, State: Initialized, Role: FOLLOWER
I20260925 00:51:38.338101 17169 consensus_queue.cc:260] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } }
Sep 25 00:51:38 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:51:38.339160 17169 ts_tablet_manager.cc:1428] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:51:38.343987 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.330939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:33565 (local address 127.0.249.126:39319)
0925 00:51:38.331076 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.331080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.331627 (+ 547us) server_negotiation.cc:407] Connection header received
0925 00:51:38.332564 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.332568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.332629 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.332737 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:38.333999 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.334623 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.335826 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.336082 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.336199 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.339735 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:38.339740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:38.339745 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:38.339782 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:38.341731 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.342223 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.342228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.342230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.342341 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.342722 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.342728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.342732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.343012 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:38.343126 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.343592 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.343753 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:51:38.344658 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.331046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:33565)
0925 00:51:38.331477 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:38.331518 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:38.332356 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:38.332896 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:38.332903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:38.333277 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:38.333843 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.333858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.334780 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.334783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.335685 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.335695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.336338 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.336343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.336567 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.337685 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:38.337704 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:38.339516 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:38.341849 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.341853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.341864 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.342117 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.342442 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.342444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.342445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.342616 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.343127 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:38.343131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:38.343348 (+ 217us) client_negotiation.cc:769] Sending connection context
0925 00:51:38.343503 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 00:51:38.343787 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260925 00:51:38.345850 17278 heartbeater.cc:347] Connected to a master server at 127.0.249.126:39319
I20260925 00:51:38.346103 17278 heartbeater.cc:464] Registering TS with master...
I20260925 00:51:38.346570 17278 heartbeater.cc:511] Master 127.0.249.126:39319 requested a full tablet report, sending...
I20260925 00:51:38.348184 16950 ts_manager.cc:194] Registered new tserver with Master: 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413)
I20260925 00:51:38.349458 16950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:33565
I20260925 00:51:38.351660 17278 heartbeater.cc:503] Master 127.0.249.126:39319 was elected leader, sending a full tablet report...
Sep 25 00:51:38 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.66@KRBTEST.COM
I20260925 00:51:38.358853 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.349079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:43413 (local address 127.0.0.1:60844)
0925 00:51:38.349242 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:38.349255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:38.349323 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:38.350020 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:38.350023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:38.350207 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:38.350313 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.350320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.351623 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.351629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.352538 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.352548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.353291 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.353294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.353424 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.353872 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:38.353887 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:38.355236 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:38.357199 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.357204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.357207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.357462 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.357782 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.357786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.357789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.357877 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.358321 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:38.358325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:38.358417 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 00:51:38.358544 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 00:51:38.358685 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 00:51:38.358927 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.349181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.0.249.66:43413)
0925 00:51:38.349580 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.349592 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.349613 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:51:38.349677 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.349684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.349884 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.350021 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:38.350436 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.351483 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.352666 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.353180 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.353269 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.355373 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:38.355383 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:38.355396 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:38.355425 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:38.357081 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.357597 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.357609 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.357611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.357686 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.357966 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.357971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.357973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.358211 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:38.358352 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.358618 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.358768 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":121,"threads_started":1}
I20260925 00:51:38.360077 17211 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a8baeb996a140a384f642e1a792db3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:51:17 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:60844
I20260925 00:51:38.361238 17284 tablet_replica.cc:330] stopping tablet replica
I20260925 00:51:38.361653 17284 raft_consensus.cc:2229] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:51:38.361936 17284 raft_consensus.cc:2258] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:51:38.362952 17284 ts_tablet_manager.cc:1905] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:51:38.367074 17284 ts_tablet_manager.cc:1918] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:51:38.367172 17284 log.cc:1196] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/0a8baeb996a140a384f642e1a792db3b
I20260925 00:51:38.367545 17284 ts_tablet_manager.cc:1939] T 0a8baeb996a140a384f642e1a792db3b P 4f71c26c19624a9ca679a0b95778638c: Deleting consensus metadata
I20260925 00:51:38.369225 16936 catalog_manager.cc:4841] TS 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413): tablet 0a8baeb996a140a384f642e1a792db3b (table test-table [id=1552d62d9a4c4496aa6d323a794da5db]) successfully deleted
W20260925 00:51:38.450838 17281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:38.451130 17281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:38.451205 17281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:38.455180 17281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:38.455259 17281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:38.455344 17281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:51:38.460621 17281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:33331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=46169
--tserver_master_addrs=127.0.249.126:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:38.461760 17281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:38.462845 17281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:38.473129 17290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:38.473173 17292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:38.473356 17281 server_base.cc:1034] running on GCE node
W20260925 00:51:38.473585 17289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:38.474052 17281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:38.474651 17281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:38.475835 17281 hybrid_clock.cc:648] HybridClock initialized: now 1790297498475818 us; error 29 us; skew 500 ppm
Sep 25 00:51:38 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:38.478706 17281 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:51:38.479521 17281 webserver.cc:460] Webserver started at http://127.0.249.67:46169/ using document root <none> and password file <none>
I20260925 00:51:38.480058 17281 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:38.480160 17281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:38.484251 17281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:51:38.486616 17299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:38.488031 17281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 00:51:38.488171 17281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "242a18419445472dacd325f2a983bcea"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "7b1ac761732966b4d9b9d00a7be1bfa3"
server_key_iv: "b42933354e5e0fb47759af99e5d46971"
server_key_version: "encryptionkey@0"
I20260925 00:51:38.488802 17281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:38.510275 17281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:38.514065 17281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:38.514293 17281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:38.514945 17281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:38.515853 17281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:51:38.515929 17281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:38.516007 17281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:51:38.516060 17281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:38.525326 17281 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:33331
I20260925 00:51:38.525349 17412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:33331 every 8 connection(s)
I20260925 00:51:38.526415 17281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:51:38.531791 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 17281
Sep 25 00:51:38 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:51:38.541846 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.532455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.0.249.126:39319)
0925 00:51:38.532700 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.532705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.532890 (+ 185us) server_negotiation.cc:407] Connection header received
0925 00:51:38.533023 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.533025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.533062 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.533136 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:38.534124 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.535001 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.536221 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.536458 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.536649 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.538243 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:38.538247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:38.538250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:38.538293 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:38.540000 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.540511 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.540516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.540517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.540601 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.540895 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.540897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.540899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.541142 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:38.541344 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.541509 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.541678 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260925 00:51:38.542009 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.528289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:47163 (local address 127.0.249.126:39319)
0925 00:51:38.528476 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.528480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.529267 (+ 787us) server_negotiation.cc:407] Connection header received
0925 00:51:38.530551 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.530553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.530600 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.530675 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:38.531917 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.532603 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.534388 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.534758 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.534873 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.537778 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:38.537781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:38.537783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:38.537803 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:38.539549 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.540173 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.540177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.540178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.540259 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.540642 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.540644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.540646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.540893 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:38.540990 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.541676 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.541790 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 00:51:38.542897 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.528652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:47163)
0925 00:51:38.529116 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:38.529158 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:38.530362 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:38.530875 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:38.530882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:38.531226 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:51:38.531751 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.531766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.532764 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.532768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.534232 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.534241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.534906 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.534909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.535040 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.535734 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:51:38.535753 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:51:38.537599 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:51:38.539674 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.539682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.539700 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.540043 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.540355 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:51:38.540357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:51:38.540358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:51:38.540528 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:51:38.541008 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:51:38.541014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:51:38.541365 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 00:51:38.541593 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 00:51:38.541914 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260925 00:51:38.544003 17413 heartbeater.cc:347] Connected to a master server at 127.0.249.126:39319
I20260925 00:51:38.544247 17413 heartbeater.cc:464] Registering TS with master...
I20260925 00:51:38.544744 17413 heartbeater.cc:511] Master 127.0.249.126:39319 requested a full tablet report, sending...
I20260925 00:51:38.545727 16950 ts_manager.cc:194] Registered new tserver with Master: 242a18419445472dacd325f2a983bcea (127.0.249.67:33331)
I20260925 00:51:38.546262 16950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:47163
I20260925 00:51:38.555824 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:51:38.560595 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.556549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.0.249.126:39319)
0925 00:51:38.556714 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.556720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.556902 (+ 182us) server_negotiation.cc:407] Connection header received
0925 00:51:38.557151 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.557154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.557258 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.557373 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:51:38.557694 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.558259 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.559242 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.559510 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.559643 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.559997 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:51:38.560145 (+ 148us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 00:51:38.560436 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47126: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 00:51:38.560662 17416 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47126: token expired
W20260925 00:51:38.560984 16876 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.249.126:39319)
I20260925 00:51:38.568186 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.561342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.0.249.126:39319)
0925 00:51:38.561482 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.561485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.561518 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:51:38.561612 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.561614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.561650 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.561716 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:51:38.562325 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.562848 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.563569 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.563821 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.563894 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.565182 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:51:38.565185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:51:38.565186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:51:38.565207 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:51:38.566750 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.567140 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.567142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.567144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.567220 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:51:38.567451 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:51:38.567453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:51:38.567454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:51:38.567650 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:51:38.567747 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.567930 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.568018 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 00:51:38.570730 16950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:51:38.574234 16950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:51:38.581681 17072 tablet_service.cc:1467] Processing CreateTablet for tablet e60d80fc94194d668358b6333b5897b8 (DEFAULT_TABLE table=test-table [id=d91d078c2eaf4eeb865f9b27b3703801]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:51:38.581831 17211 tablet_service.cc:1467] Processing CreateTablet for tablet e60d80fc94194d668358b6333b5897b8 (DEFAULT_TABLE table=test-table [id=d91d078c2eaf4eeb865f9b27b3703801]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:51:38.582408 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60d80fc94194d668358b6333b5897b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:51:38.582497 17211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60d80fc94194d668358b6333b5897b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:51:38.585168 17030 tablet_bootstrap.cc:492] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Bootstrap starting.
I20260925 00:51:38.586126 17030 tablet_bootstrap.cc:654] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Neither blocks nor log segments found. Creating new log.
I20260925 00:51:38.586833 17169 tablet_bootstrap.cc:492] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: Bootstrap starting.
I20260925 00:51:38.587507 17030 tablet_bootstrap.cc:492] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: No bootstrap required, opened a new log
I20260925 00:51:38.587604 17030 ts_tablet_manager.cc:1397] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 00:51:38.588184 17169 tablet_bootstrap.cc:654] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: Neither blocks nor log segments found. Creating new log.
I20260925 00:51:38.588107 17030 raft_consensus.cc:348] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:38.588236 17030 raft_consensus.cc:374] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:51:38.588276 17030 raft_consensus.cc:729] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f23e7ef362c46e08c6708bbf8afd2c8, State: Initialized, Role: FOLLOWER
I20260925 00:51:38.588429 17030 consensus_queue.cc:260] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:38.588758 17030 ts_tablet_manager.cc:1428] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 00:51:38.589452 17169 tablet_bootstrap.cc:492] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: No bootstrap required, opened a new log
I20260925 00:51:38.589540 17169 ts_tablet_manager.cc:1397] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 00:51:38.589977 17169 raft_consensus.cc:348] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:38.590095 17169 raft_consensus.cc:374] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:51:38.590152 17169 raft_consensus.cc:729] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f71c26c19624a9ca679a0b95778638c, State: Initialized, Role: FOLLOWER
I20260925 00:51:38.590282 17169 consensus_queue.cc:260] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:38.590583 17169 ts_tablet_manager.cc:1428] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 00:51:38.591984 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.580822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:33331 (local address 127.0.0.1:58410)
0925 00:51:38.580993 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:38.581008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:38.581087 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:38.582031 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:38.582034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:38.582052 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:51:38.582158 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.582164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.583734 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.583737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.585026 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.585041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.591517 (+ 6476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.591526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.591621 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.591667 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 00:51:38.591766 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 00:51:38.591859 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 00:51:38.592154 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.580991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.0.249.67:33331)
0925 00:51:38.581474 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.581486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.581510 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:51:38.581587 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.581602 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.581830 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.582334 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:51:38.582499 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.583573 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.588777 (+ 5204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.591390 (+ 2613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.591519 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.591698 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.591906 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.592015 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":189,"threads_started":1}
I20260925 00:51:38.595327 17347 tablet_service.cc:1467] Processing CreateTablet for tablet e60d80fc94194d668358b6333b5897b8 (DEFAULT_TABLE table=test-table [id=d91d078c2eaf4eeb865f9b27b3703801]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:51:38.596388 17347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60d80fc94194d668358b6333b5897b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:51:38.603057 17425 tablet_bootstrap.cc:492] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: Bootstrap starting.
I20260925 00:51:38.605037 17425 tablet_bootstrap.cc:654] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: Neither blocks nor log segments found. Creating new log.
I20260925 00:51:38.606161 17425 log.cc:824] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:38.607916 17425 tablet_bootstrap.cc:492] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: No bootstrap required, opened a new log
I20260925 00:51:38.608129 17425 ts_tablet_manager.cc:1397] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 00:51:38.610966 17425 raft_consensus.cc:348] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:38.611227 17425 raft_consensus.cc:374] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:51:38.611305 17425 raft_consensus.cc:729] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242a18419445472dacd325f2a983bcea, State: Initialized, Role: FOLLOWER
I20260925 00:51:38.611756 17425 consensus_queue.cc:260] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:38.612430 17413 heartbeater.cc:503] Master 127.0.249.126:39319 was elected leader, sending a full tablet report...
I20260925 00:51:38.612627 17425 ts_tablet_manager.cc:1428] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20260925 00:51:38.628768 17139 tablet.cc:2369] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:51:38.738337 17427 raft_consensus.cc:482] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:51:38.738562 17427 raft_consensus.cc:504] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:38.739740 17427 leader_election.cc:288] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413), 4f23e7ef362c46e08c6708bbf8afd2c8 (127.0.249.65:32805)
I20260925 00:51:38.744012 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.739970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:43413 (local address 127.0.249.67:46411)
0925 00:51:38.740129 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:38.740144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:38.740236 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:38.740607 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:38.740610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:38.740629 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:51:38.740800 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.740809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.741748 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.741751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.742644 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.742652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.743576 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.743580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.743687 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.743714 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:51:38.743794 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 00:51:38.743866 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 00:51:38.744057 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.740046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:46411 (local address 127.0.249.66:43413)
0925 00:51:38.740202 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.740207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.740221 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:51:38.740375 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.740380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.740476 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.740581 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:51:38.740920 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.741640 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.742810 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.743451 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.743566 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.743686 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.743827 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.743892 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 00:51:38.744565 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.740099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:55333 (local address 127.0.249.65:32805)
0925 00:51:38.740395 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.740401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.740723 (+ 322us) server_negotiation.cc:407] Connection header received
0925 00:51:38.740837 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.740843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.740922 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.741017 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:51:38.741327 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.741999 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.743301 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.743912 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.744026 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.744194 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.744373 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.744457 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
I20260925 00:51:38.744577 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.739986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:32805 (local address 127.0.249.67:55333)
0925 00:51:38.740616 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:38.740630 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:38.740726 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:38.741057 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:51:38.741060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:51:38.741077 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:51:38.741214 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.741221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.742140 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.742142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.743156 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:51:38.743168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.744072 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.744079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:51:38.744197 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.744231 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 00:51:38.744333 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 00:51:38.744424 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":301,"thread_start_us":94,"threads_started":1}
I20260925 00:51:38.744895 17231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60d80fc94194d668358b6333b5897b8" candidate_uuid: "242a18419445472dacd325f2a983bcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f71c26c19624a9ca679a0b95778638c" is_pre_election: true
I20260925 00:51:38.745147 17092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60d80fc94194d668358b6333b5897b8" candidate_uuid: "242a18419445472dacd325f2a983bcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" is_pre_election: true
I20260925 00:51:38.745288 17231 raft_consensus.cc:2454] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 242a18419445472dacd325f2a983bcea in term 0.
I20260925 00:51:38.745486 17092 raft_consensus.cc:2454] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 242a18419445472dacd325f2a983bcea in term 0.
I20260925 00:51:38.745934 17300 leader_election.cc:302] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242a18419445472dacd325f2a983bcea, 4f23e7ef362c46e08c6708bbf8afd2c8; no voters:
I20260925 00:51:38.746213 17427 raft_consensus.cc:2781] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:51:38.746305 17427 raft_consensus.cc:482] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:51:38.746341 17427 raft_consensus.cc:3037] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:51:38.747541 17427 raft_consensus.cc:504] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:38.747949 17427 leader_election.cc:288] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [CANDIDATE]: Term 1 election: Requested vote from peers 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413), 4f23e7ef362c46e08c6708bbf8afd2c8 (127.0.249.65:32805)
I20260925 00:51:38.748272 17092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60d80fc94194d668358b6333b5897b8" candidate_uuid: "242a18419445472dacd325f2a983bcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8"
I20260925 00:51:38.748296 17231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60d80fc94194d668358b6333b5897b8" candidate_uuid: "242a18419445472dacd325f2a983bcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f71c26c19624a9ca679a0b95778638c"
I20260925 00:51:38.748420 17092 raft_consensus.cc:3037] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:51:38.748425 17231 raft_consensus.cc:3037] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:51:38.749508 17092 raft_consensus.cc:2454] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242a18419445472dacd325f2a983bcea in term 1.
I20260925 00:51:38.749516 17231 raft_consensus.cc:2454] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242a18419445472dacd325f2a983bcea in term 1.
I20260925 00:51:38.749830 17302 leader_election.cc:302] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242a18419445472dacd325f2a983bcea, 4f71c26c19624a9ca679a0b95778638c; no voters:
I20260925 00:51:38.749989 17427 raft_consensus.cc:2781] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:51:38.750231 17427 raft_consensus.cc:686] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 1 LEADER]: Becoming Leader. State: Replica: 242a18419445472dacd325f2a983bcea, State: Running, Role: LEADER
I20260925 00:51:38.750558 17427 consensus_queue.cc:237] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:38.753672 16949 catalog_manager.cc:5495] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea reported cstate change: term changed from 0 to 1, leader changed from <none> to 242a18419445472dacd325f2a983bcea (127.0.249.67). New cstate: current_term: 1 leader_uuid: "242a18419445472dacd325f2a983bcea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } health_report { overall_health: UNKNOWN } } }
I20260925 00:51:38.776257 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:38.772670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.0.249.67:33331)
0925 00:51:38.772817 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:51:38.772822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:51:38.772836 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:51:38.773012 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:51:38.773015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:51:38.773068 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:51:38.773181 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:51:38.773445 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.773955 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.774798 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:51:38.775021 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:51:38.775153 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:51:38.775466 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:51:38.775868 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:51:38.776005 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 00:51:38.776041 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 00:51:38.776108 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 00:51:38.777434 17414 tablet.cc:2369] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:51:38.782289 17231 raft_consensus.cc:1261] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Refusing update from remote peer 242a18419445472dacd325f2a983bcea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:51:38.782305 17092 raft_consensus.cc:1261] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Refusing update from remote peer 242a18419445472dacd325f2a983bcea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:51:38.783128 17427 consensus_queue.cc:1035] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:51:38.783368 17430 consensus_queue.cc:1035] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:51:38.790716 17435 mvcc.cc:205] Tried to move back new op lower bound from 7333058555005235200 to 7333058554885136384. Current Snapshot: MvccSnapshot[applied={T|T < 7333058555005235200}]
I20260925 00:51:38.798219 16950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:51:38.798360 16950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:51:38.800707 16950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Sending DeleteTablet for 3 replicas of tablet e60d80fc94194d668358b6333b5897b8
I20260925 00:51:38.801326 17211 tablet_service.cc:1514] Processing DeleteTablet for tablet e60d80fc94194d668358b6333b5897b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:51:38 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:60844
I20260925 00:51:38.801383 17347 tablet_service.cc:1514] Processing DeleteTablet for tablet e60d80fc94194d668358b6333b5897b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:51:38 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:58410
I20260925 00:51:38.801594 17284 tablet_replica.cc:330] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: stopping tablet replica
I20260925 00:51:38.801723 17284 raft_consensus.cc:2229] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:51:38.801847 17284 raft_consensus.cc:2258] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:51:38.801858 17443 tablet_replica.cc:330] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: stopping tablet replica
I20260925 00:51:38.801836 17072 tablet_service.cc:1514] Processing DeleteTablet for tablet e60d80fc94194d668358b6333b5897b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:51:38 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:53388
I20260925 00:51:38.802163 17443 raft_consensus.cc:2229] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:51:38.802393 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 16988
I20260925 00:51:38.802508 17284 ts_tablet_manager.cc:1905] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:51:38.803361 17443 raft_consensus.cc:2258] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:51:38.804986 17284 ts_tablet_manager.cc:1918] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:51:38.805069 17284 log.cc:1196] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/e60d80fc94194d668358b6333b5897b8
I20260925 00:51:38.805337 17284 ts_tablet_manager.cc:1939] T e60d80fc94194d668358b6333b5897b8 P 4f71c26c19624a9ca679a0b95778638c: Deleting consensus metadata
I20260925 00:51:38.810582 16936 catalog_manager.cc:4841] TS 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413): tablet e60d80fc94194d668358b6333b5897b8 (table test-table [id=d91d078c2eaf4eeb865f9b27b3703801]) successfully deleted
I20260925 00:51:38.810930 17443 ts_tablet_manager.cc:1905] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:51:38.812140 16934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:32805 (error 108)
W20260925 00:51:38.813349 17145 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:51:38.813017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:32805 (local address 127.0.0.1:53390)
0925 00:51:38.813172 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:38.813249 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:32805: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 00:51:38.813500 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 17143
I20260925 00:51:38.813532 17443 ts_tablet_manager.cc:1918] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:51:38.813625 17443 log.cc:1196] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/e60d80fc94194d668358b6333b5897b8
I20260925 00:51:38.813921 17443 ts_tablet_manager.cc:1939] T e60d80fc94194d668358b6333b5897b8 P 242a18419445472dacd325f2a983bcea: Deleting consensus metadata
I20260925 00:51:38.814983 16937 catalog_manager.cc:4841] TS 242a18419445472dacd325f2a983bcea (127.0.249.67:33331): tablet e60d80fc94194d668358b6333b5897b8 (table test-table [id=d91d078c2eaf4eeb865f9b27b3703801]) successfully deleted
W20260925 00:51:38.818714 16934 catalog_manager.cc:4584] TS 4f23e7ef362c46e08c6708bbf8afd2c8 (127.0.249.65:32805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e60d80fc94194d668358b6333b5897b8: Network error: Client connection negotiation failed: client connection to 127.0.249.65:32805: connect: Connection reset by peer (error 104)
I20260925 00:51:38.820477 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 17281
I20260925 00:51:38.826025 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 16917
I20260925 00:51:58.832607 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39319
--webserver_interface=127.0.249.126
--webserver_port=43739
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:51:58.944530 17446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:58.944852 17446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:58.944957 17446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:58.949067 17446 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 00:51:58.949153 17446 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 00:51:58.949186 17446 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 00:51:58.949239 17446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:51:58.949301 17446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:58.949348 17446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:58.949404 17446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:51:58.949445 17446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:51:58.954829 17446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:39319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:58.955925 17446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:58.957026 17446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:51:58.965905 17446 server_base.cc:1034] running on GCE node
W20260925 00:51:58.965869 17452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:58.965855 17454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:58.965837 17451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:58.966656 17446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:58.967674 17446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:58.968825 17446 hybrid_clock.cc:648] HybridClock initialized: now 1790297518968786 us; error 54 us; skew 500 ppm
Sep 25 00:51:58 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:58.971552 17446 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:51:58.972366 17446 webserver.cc:460] Webserver started at http://127.0.249.126:43739/ using document root <none> and password file <none>
I20260925 00:51:58.972934 17446 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:58.973026 17446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:58.976758 17446 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 00:51:58.978951 17461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:58.980185 17446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:51:58.980319 17446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "94fd4814158f4098892f59619919fa63"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "65367f80b3d2bdf200bd0f3ec5c7442e"
server_key_iv: "5af91dc9d1f6c79a36fd32e922540e46"
server_key_version: "encryptionkey@0"
I20260925 00:51:58.980890 17446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:58.999285 17446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:59.002215 17446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:59.002413 17446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:59.008965 17446 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:39319
I20260925 00:51:59.009269 17513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:39319 every 8 connection(s)
I20260925 00:51:59.010053 17446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:51:59.014292 17514 sys_catalog.cc:263] Verifying existing consensus state
I20260925 00:51:59.015890 17514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Bootstrap starting.
I20260925 00:51:59.018890 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 17446
I20260925 00:51:59.019318 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:32805
--local_ip_for_outbound_sockets=127.0.249.65
--tserver_master_addrs=127.0.249.126:39319
--webserver_port=42481
--webserver_interface=127.0.249.65
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:51:59.032950 17514 log.cc:824] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:59.044020 17514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:51:59.044409 17514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: Bootstrap complete.
I20260925 00:51:59.048413 17514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:59.048825 17514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fd4814158f4098892f59619919fa63, State: Initialized, Role: FOLLOWER
I20260925 00:51:59.049338 17514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:59.049552 17514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:51:59.049643 17514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:51:59.049741 17514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 2 FOLLOWER]: Advancing to term 3
I20260925 00:51:59.051587 17514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:59.051908 17514 leader_election.cc:302] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94fd4814158f4098892f59619919fa63; no voters:
I20260925 00:51:59.052227 17514 leader_election.cc:288] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 00:51:59.052333 17518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 00:51:59.052538 17518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [term 3 LEADER]: Becoming Leader. State: Replica: 94fd4814158f4098892f59619919fa63, State: Running, Role: LEADER
I20260925 00:51:59.052917 17518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } }
I20260925 00:51:59.053479 17514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:51:59.054382 17520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94fd4814158f4098892f59619919fa63. Latest consensus state: current_term: 3 leader_uuid: "94fd4814158f4098892f59619919fa63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } } }
I20260925 00:51:59.054502 17520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: This master's current role is: LEADER
I20260925 00:51:59.054437 17519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "94fd4814158f4098892f59619919fa63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fd4814158f4098892f59619919fa63" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 39319 } } }
I20260925 00:51:59.054553 17519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63 [sys.catalog]: This master's current role is: LEADER
I20260925 00:51:59.058339 17533 catalog_manager.cc:1259] Loaded cluster ID: f56ec958673149759dbd815dc051b1ad
I20260925 00:51:59.058414 17533 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: loading cluster ID for follower catalog manager: success
I20260925 00:51:59.060135 17533 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94fd4814158f4098892f59619919fa63: acquiring CA information for follower catalog manager: success
F20260925 00:51:59.061046 17533 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f094e4ac9ad google::LogMessage::Fail() at ??:0
@ 0x7f094e4af88a google::LogMessage::SendToLog() at ??:0
@ 0x7f094e4ac4de google::LogMessage::Flush() at ??:0
@ 0x7f094e4ada19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f095117018d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0958e11989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f0958dee023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f0958dee54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f0958de7652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f0958de69e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f0958e2d541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f09506ef512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0951d336db start_thread at ??:0
@ 0x7f094cfaa61f clone at ??:0
W20260925 00:51:59.153925 17516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:59.154184 17516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:59.154228 17516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:59.158377 17516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:59.158439 17516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:59.158514 17516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:51:59.163729 17516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:32805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=42481
--tserver_master_addrs=127.0.249.126:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:59.164822 17516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:59.165894 17516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:59.175619 17555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:59.175848 17516 server_base.cc:1034] running on GCE node
W20260925 00:51:59.176038 17554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:59.176038 17557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:59.176493 17516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:59.177158 17516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:59.178339 17516 hybrid_clock.cc:648] HybridClock initialized: now 1790297519178316 us; error 76 us; skew 500 ppm
Sep 25 00:51:59 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:59.181171 17516 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:51:59.182034 17516 webserver.cc:460] Webserver started at http://127.0.249.65:42481/ using document root <none> and password file <none>
I20260925 00:51:59.182590 17516 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:59.182690 17516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:59.186748 17516 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 00:51:59.188957 17564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:59.190353 17516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 00:51:59.190488 17516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "4f23e7ef362c46e08c6708bbf8afd2c8"
format_stamp: "Formatted at 2026-09-25 00:51:16 on dist-test-slave-05m0"
server_key: "1289cae27b711e540c4fd6ab351bf78c"
server_key_iv: "0381d72f751d0b235ea12a84981087d8"
server_key_version: "encryptionkey@0"
I20260925 00:51:59.191013 17516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:59.203559 17516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:59.206700 17516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:59.206892 17516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:59.207446 17516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:59.208568 17571 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 00:51:59.214088 17516 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 00:51:59.214171 17516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260925 00:51:59.214250 17516 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 00:51:59.215963 17516 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 00:51:59.216032 17516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 00:51:59.216364 17571 tablet_bootstrap.cc:492] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Bootstrap starting.
I20260925 00:51:59.226568 17516 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:32805
I20260925 00:51:59.227494 17516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:51:59.228037 17678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:32805 every 8 connection(s)
I20260925 00:51:59.236084 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 17516
I20260925 00:51:59.236482 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:43413
--local_ip_for_outbound_sockets=127.0.249.66
--tserver_master_addrs=127.0.249.126:39319
--webserver_port=37945
--webserver_interface=127.0.249.66
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:51:59.238289 17571 log.cc:824] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Log is configured to *not* fsync() on all Append() calls
I20260925 00:51:59.244985 17571 tablet_bootstrap.cc:492] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 00:51:59.245332 17571 tablet_bootstrap.cc:492] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Bootstrap complete.
I20260925 00:51:59.246428 17571 ts_tablet_manager.cc:1397] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.004s
I20260925 00:51:59.248769 17571 raft_consensus.cc:348] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:59.249202 17571 raft_consensus.cc:729] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f23e7ef362c46e08c6708bbf8afd2c8, State: Initialized, Role: FOLLOWER
I20260925 00:51:59.249706 17571 consensus_queue.cc:260] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:51:59.250639 17571 ts_tablet_manager.cc:1428] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260925 00:51:59.273005 17681 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:51:59.229564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:52893)
0925 00:51:59.230024 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:59.230075 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 00:51:59.230893 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:51:59.272240 (+ 41347us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
W20260925 00:51:59.273634 17679 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:39319 (0 consecutive failures): Network error: Failed to ping master at 127.0.249.126:39319: Client connection negotiation failed: client connection to 127.0.249.126:39319: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 00:51:59.274351 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:59.274095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:58793)
0925 00:51:59.274227 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:59.274299 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 00:51:59.275086 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:59.274846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:43449)
0925 00:51:59.274968 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:59.275025 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 00:51:59.275283 17679 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 00:51:59.358171 17682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:59.358429 17682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:59.358500 17682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:59.362635 17682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:59.362696 17682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:59.362782 17682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:51:59.368117 17682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:43413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=37945
--tserver_master_addrs=127.0.249.126:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:59.369149 17682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:59.370185 17682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:59.379608 17689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:59.379655 17690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:59.379655 17692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:59.379887 17682 server_base.cc:1034] running on GCE node
I20260925 00:51:59.380280 17682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:59.380879 17682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:59.382052 17682 hybrid_clock.cc:648] HybridClock initialized: now 1790297519382035 us; error 29 us; skew 500 ppm
Sep 25 00:51:59 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:59.387548 17682 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:51:59.388381 17682 webserver.cc:460] Webserver started at http://127.0.249.66:37945/ using document root <none> and password file <none>
I20260925 00:51:59.388926 17682 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:59.389029 17682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:59.392747 17682 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:51:59.394996 17699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:59.396195 17682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:51:59.396314 17682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "4f71c26c19624a9ca679a0b95778638c"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "f4e6220cd030abb7b9c9aad04e0ce58f"
server_key_iv: "b88edebb4a808492f05453c56b8435cf"
server_key_version: "encryptionkey@0"
I20260925 00:51:59.396859 17682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:59.408757 17682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:59.411860 17682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:59.412078 17682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:59.412688 17682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:59.413623 17682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:51:59.413704 17682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:59.413796 17682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:51:59.413856 17682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:59.423483 17682 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:43413
I20260925 00:51:59.423504 17812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:43413 every 8 connection(s)
I20260925 00:51:59.424606 17682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:51:59.428503 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:59.426718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:58267)
0925 00:51:59.427185 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:59.427297 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
W20260925 00:51:59.428972 17813 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:39319 (0 consecutive failures): Network error: Failed to ping master at 127.0.249.126:39319: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
I20260925 00:51:59.429661 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:59.429363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:33897)
0925 00:51:59.429514 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:59.429576 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 00:51:59.430392 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:59.430185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:48125)
0925 00:51:59.430297 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:59.430341 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 00:51:59.430583 17813 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 00:51:59.433997 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 17682
I20260925 00:51:59.434383 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:33331
--local_ip_for_outbound_sockets=127.0.249.67
--tserver_master_addrs=127.0.249.126:39319
--webserver_port=46169
--webserver_interface=127.0.249.67
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:51:59.543828 17816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:51:59.544121 17816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:51:59.544195 17816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:51:59.548200 17816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:51:59.548287 17816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:51:59.548374 17816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:51:59.553653 17816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:33331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=46169
--tserver_master_addrs=127.0.249.126:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:51:59.554754 17816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:51:59.555784 17816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:51:59.565513 17824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:51:59.565518 17821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:59.565640 17816 server_base.cc:1034] running on GCE node
W20260925 00:51:59.565825 17822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:51:59.566216 17816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:51:59.566855 17816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:51:59.568012 17816 hybrid_clock.cc:648] HybridClock initialized: now 1790297519567980 us; error 49 us; skew 500 ppm
Sep 25 00:51:59 dist-test-slave-05m0 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:51:59.570626 17816 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:51:59.571430 17816 webserver.cc:460] Webserver started at http://127.0.249.67:46169/ using document root <none> and password file <none>
I20260925 00:51:59.571964 17816 fs_manager.cc:359] Metadata directory not provided
I20260925 00:51:59.572074 17816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:51:59.575837 17816 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:51:59.577893 17831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:59.579118 17816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:51:59.579231 17816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "242a18419445472dacd325f2a983bcea"
format_stamp: "Formatted at 2026-09-25 00:51:17 on dist-test-slave-05m0"
server_key: "7b1ac761732966b4d9b9d00a7be1bfa3"
server_key_iv: "b42933354e5e0fb47759af99e5d46971"
server_key_version: "encryptionkey@0"
I20260925 00:51:59.579721 17816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:51:59.592808 17816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:51:59.595960 17816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:51:59.596155 17816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:51:59.596724 17816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:51:59.597535 17816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:51:59.597581 17816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:59.597656 17816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:51:59.597689 17816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:51:59.607028 17816 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:33331
I20260925 00:51:59.607036 17944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:33331 every 8 connection(s)
I20260925 00:51:59.608078 17816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:51:59.610333 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 17816
I20260925 00:51:59.612264 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:59.610283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:59829)
0925 00:51:59.610825 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:59.610943 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
W20260925 00:51:59.612730 17945 heartbeater.cc:644] Failed to heartbeat to 127.0.249.126:39319 (0 consecutive failures): Network error: Failed to ping master at 127.0.249.126:39319: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
I20260925 00:51:59.613322 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:59.613110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:35051)
0925 00:51:59.613219 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:59.613269 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 00:51:59.613929 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:51:59.613755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:52875)
0925 00:51:59.613847 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 00:51:59.613886 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 00:51:59.614100 17945 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 00:52:00.276427 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:00.275951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:43601)
0925 00:52:00.276254 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:00.276314 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260925 00:52:00.431468 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:00.431089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:43195)
0925 00:52:00.431317 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:00.431370 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260925 00:52:00.615029 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:00.614606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:54561)
0925 00:52:00.614866 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:00.614923 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260925 00:52:00.968276 17952 raft_consensus.cc:482] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:00.968475 17952 raft_consensus.cc:504] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:52:00.969676 17952 leader_election.cc:288] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 242a18419445472dacd325f2a983bcea (127.0.249.67:33331), 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413)
Sep 25 00:52:00 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.66@KRBTEST.COM
Sep 25 00:52:00 dist-test-slave-05m0 krb5kdc[16369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.67@KRBTEST.COM
I20260925 00:52:00.985327 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:00.969980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:43413 (local address 127.0.249.65:46787)
0925 00:52:00.970520 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:00.970534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:00.970618 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:00.972378 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:00.972383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:00.973087 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:00.973595 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:00.973611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:00.974977 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:00.974980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:00.975886 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:00.975895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:00.977088 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:00.977092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:00.977276 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:00.977971 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:00.978012 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:00.979916 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:00.982781 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:00.982786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:00.982801 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:00.983063 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:00.983453 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:00.983456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:00.983458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:00.983596 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:00.984424 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:00.984430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:00.984663 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 00:52:00.984847 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 00:52:00.985133 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":1,"spinlock_wait_cycles":17664,"thread_start_us":82,"threads_started":1}
I20260925 00:52:00.985384 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:00.970472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:46787 (local address 127.0.249.66:43413)
0925 00:52:00.970874 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:00.970887 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:00.970924 (+ 37us) server_negotiation.cc:407] Connection header received
0925 00:52:00.971707 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:00.971713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:00.972012 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:00.972708 (+ 696us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:00.973749 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:00.974821 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:00.976008 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:00.976906 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:00.977136 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:00.980124 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:00.980148 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:00.980167 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:00.980241 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:00.982675 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:00.983186 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:00.983191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:00.983197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:00.983331 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:00.983696 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:00.983701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:00.983703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:00.984324 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:00.984442 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:00.984959 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 00:52:00.985224 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260925 00:52:00.986205 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:00.969834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:33331 (local address 127.0.249.65:41233)
0925 00:52:00.970114 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:00.970129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:00.970233 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:00.972381 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:00.972386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:00.973090 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:00.973638 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:00.973652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:00.975811 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:00.975814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:00.976399 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:00.976409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:00.977864 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:00.977868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:00.977955 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:00.978456 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:00.978472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:00.980468 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:00.983977 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:00.983980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:00.983982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:00.984207 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:00.984635 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:00.984641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:00.984645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:00.984752 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:00.985583 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:00.985588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:00.985702 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 00:52:00.985796 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 00:52:00.985948 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":23,"spinlock_wait_cycles":15616,"thread_start_us":58,"threads_started":1}
I20260925 00:52:00.987193 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:00.970034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:41233 (local address 127.0.249.67:33331)
0925 00:52:00.970469 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:00.970495 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:00.970540 (+ 45us) server_negotiation.cc:407] Connection header received
0925 00:52:00.971459 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:00.971468 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:00.971918 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:00.972282 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:00.973842 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:00.975645 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:00.976533 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:00.977490 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:00.977673 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:00.980997 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:00.981006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:00.981026 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:00.981068 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:00.983861 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:00.984323 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:00.984331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:00.984337 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:00.984490 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:00.984875 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:00.984881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:00.984884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:00.985433 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:00.986432 (+ 999us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:00.986611 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 00:52:00.986933 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":113,"threads_started":1}
W20260925 00:52:00.989336 17567 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
W20260925 00:52:00.989538 17568 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 242a18419445472dacd325f2a983bcea (127.0.249.67:33331): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
I20260925 00:52:00.989619 17568 leader_election.cc:302] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f23e7ef362c46e08c6708bbf8afd2c8; no voters: 242a18419445472dacd325f2a983bcea, 4f71c26c19624a9ca679a0b95778638c
I20260925 00:52:00.989917 17952 raft_consensus.cc:2726] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:52:01.277551 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:01.277205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:52231)
0925 00:52:01.277360 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:01.277467 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 00:52:01.432619 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:01.432198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:45627)
0925 00:52:01.432455 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:01.432511 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260925 00:52:01.616163 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:01.615754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:59709)
0925 00:52:01.615986 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:01.616051 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260925 00:52:02.278854 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:02.278390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:50489)
0925 00:52:02.278689 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:02.278758 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260925 00:52:02.433918 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:02.433508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:56953)
0925 00:52:02.433765 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:02.433828 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260925 00:52:02.617362 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:02.616939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:46185)
0925 00:52:02.617206 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:02.617271 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
I20260925 00:52:02.734236 17962 raft_consensus.cc:482] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:02.734371 17962 raft_consensus.cc:504] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:52:02.734833 17962 leader_election.cc:288] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 242a18419445472dacd325f2a983bcea (127.0.249.67:33331), 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413)
W20260925 00:52:02.735785 17568 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 242a18419445472dacd325f2a983bcea (127.0.249.67:33331): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
W20260925 00:52:02.735893 17567 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
I20260925 00:52:02.735953 17567 leader_election.cc:302] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f23e7ef362c46e08c6708bbf8afd2c8; no voters: 242a18419445472dacd325f2a983bcea, 4f71c26c19624a9ca679a0b95778638c
I20260925 00:52:02.736100 17962 raft_consensus.cc:2726] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:52:03.280102 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:03.279721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:37489)
0925 00:52:03.279955 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:03.280007 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260925 00:52:03.435027 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:03.434638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:47359)
0925 00:52:03.434884 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:03.434940 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
I20260925 00:52:03.618532 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:03.618126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:33775)
0925 00:52:03.618377 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:03.618433 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260925 00:52:04.281492 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:04.281000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:33741)
0925 00:52:04.281279 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:04.281337 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260925 00:52:04.386518 17967 raft_consensus.cc:482] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:04.386677 17967 raft_consensus.cc:504] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:52:04.387116 17967 leader_election.cc:288] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 242a18419445472dacd325f2a983bcea (127.0.249.67:33331), 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413)
W20260925 00:52:04.387771 17568 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 242a18419445472dacd325f2a983bcea (127.0.249.67:33331): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
W20260925 00:52:04.387856 17567 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
I20260925 00:52:04.387941 17567 leader_election.cc:302] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f23e7ef362c46e08c6708bbf8afd2c8; no voters: 242a18419445472dacd325f2a983bcea, 4f71c26c19624a9ca679a0b95778638c
I20260925 00:52:04.388069 17967 raft_consensus.cc:2726] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:52:04.436239 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:04.435797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:59291)
0925 00:52:04.436070 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:04.436141 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260925 00:52:04.619695 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:04.619272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:39365)
0925 00:52:04.619514 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:04.619569 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":84,"threads_started":1}
I20260925 00:52:05.282600 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:05.282211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:36883)
0925 00:52:05.282448 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:05.282506 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260925 00:52:05.437333 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:05.436953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:50309)
0925 00:52:05.437189 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:05.437242 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260925 00:52:05.620797 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:05.620413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:54243)
0925 00:52:05.620659 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:05.620711 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
I20260925 00:52:06.283836 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:06.283355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:55539)
0925 00:52:06.283677 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:06.283738 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260925 00:52:06.438525 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:06.438064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:52861)
0925 00:52:06.438373 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:06.438426 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
I20260925 00:52:06.622002 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:06.621578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:42179)
0925 00:52:06.621860 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:06.621913 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260925 00:52:07.285053 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:07.284646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:38239)
0925 00:52:07.284892 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:07.284949 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260925 00:52:07.439765 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:07.439341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:43589)
0925 00:52:07.439602 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:07.439667 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260925 00:52:07.623121 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:07.622730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:50543)
0925 00:52:07.622968 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:07.623021 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260925 00:52:08.286377 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:08.285860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:44105)
0925 00:52:08.286171 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:08.286249 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260925 00:52:08.441107 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:08.440634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:39099)
0925 00:52:08.440938 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:08.441006 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260925 00:52:08.624323 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:08.623891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:58203)
0925 00:52:08.624151 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:08.624230 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260925 00:52:08.925633 17982 raft_consensus.cc:482] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:08.925766 17982 raft_consensus.cc:504] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:52:08.926151 17982 leader_election.cc:288] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 242a18419445472dacd325f2a983bcea (127.0.249.67:33331), 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413)
W20260925 00:52:08.926893 17568 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 242a18419445472dacd325f2a983bcea (127.0.249.67:33331): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
W20260925 00:52:08.927040 17567 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
I20260925 00:52:08.927131 17567 leader_election.cc:302] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f23e7ef362c46e08c6708bbf8afd2c8; no voters: 242a18419445472dacd325f2a983bcea, 4f71c26c19624a9ca679a0b95778638c
I20260925 00:52:08.927292 17982 raft_consensus.cc:2726] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:52:09.287727 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:09.287286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:41383)
0925 00:52:09.287581 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:09.287636 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260925 00:52:09.442415 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:09.442013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:58795)
0925 00:52:09.442272 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:09.442325 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260925 00:52:09.625447 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:09.624997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:52063)
0925 00:52:09.625258 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:09.625308 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20260925 00:52:10.288916 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:10.288514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:46997)
0925 00:52:10.288770 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:10.288824 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260925 00:52:10.443652 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:10.443230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:40697)
0925 00:52:10.443492 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:10.443554 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260925 00:52:10.626576 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:10.626178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:43957)
0925 00:52:10.626433 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:10.626484 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260925 00:52:10.921023 17989 raft_consensus.cc:482] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:10.921217 17989 raft_consensus.cc:504] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "242a18419445472dacd325f2a983bcea" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 33331 } } peers { permanent_uuid: "4f71c26c19624a9ca679a0b95778638c" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 43413 } } peers { permanent_uuid: "4f23e7ef362c46e08c6708bbf8afd2c8" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 32805 } }
I20260925 00:52:10.921731 17989 leader_election.cc:288] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 242a18419445472dacd325f2a983bcea (127.0.249.67:33331), 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413)
W20260925 00:52:10.922565 17568 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 242a18419445472dacd325f2a983bcea (127.0.249.67:33331): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
W20260925 00:52:10.922710 17567 leader_election.cc:341] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f71c26c19624a9ca679a0b95778638c (127.0.249.66:43413): Not found: Tablet not found: e60d80fc94194d668358b6333b5897b8
I20260925 00:52:10.922842 17567 leader_election.cc:302] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f23e7ef362c46e08c6708bbf8afd2c8; no voters: 242a18419445472dacd325f2a983bcea, 4f71c26c19624a9ca679a0b95778638c
I20260925 00:52:10.923038 17989 raft_consensus.cc:2726] T e60d80fc94194d668358b6333b5897b8 P 4f23e7ef362c46e08c6708bbf8afd2c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 00:52:11.290320 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:11.289856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:55239)
0925 00:52:11.290155 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:11.290218 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
I20260925 00:52:11.444896 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:11.444503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:55581)
0925 00:52:11.444751 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:11.444807 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260925 00:52:11.627671 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:11.627288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:59391)
0925 00:52:11.627527 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:11.627581 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260925 00:52:12.291579 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:12.291183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:40315)
0925 00:52:12.291429 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:12.291486 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
I20260925 00:52:12.446182 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:12.445784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:40009)
0925 00:52:12.446035 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:12.446090 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260925 00:52:12.628786 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:12.628395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:37495)
0925 00:52:12.628638 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:12.628694 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260925 00:52:13.292771 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:13.292330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:40959)
0925 00:52:13.292624 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:13.292678 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
I20260925 00:52:13.447321 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:13.446910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:36561)
0925 00:52:13.447167 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:13.447222 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
I20260925 00:52:13.630028 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:13.629622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:52889)
0925 00:52:13.629884 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:13.629938 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260925 00:52:14.294620 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:14.294098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.65:56253)
0925 00:52:14.294436 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:14.294522 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260925 00:52:14.448714 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:14.448274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.66:37119)
0925 00:52:14.448539 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:14.448617 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
I20260925 00:52:14.622715 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 17516
I20260925 00:52:14.629032 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 17682
I20260925 00:52:14.631378 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:14.630949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:39319 (local address 127.0.249.67:35971)
0925 00:52:14.631238 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:14.631290 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.126:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":118,"threads_started":1}
I20260925 00:52:14.635239 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 17816
2026-09-25T00:52:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59927 ms)
[----------] 30 tests from SecurityITest (242579 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:15 dist-test-slave-05m0 krb5kdc[18004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:15 dist-test-slave-05m0 krb5kdc[18004](info): set up 2 sockets
Sep 25 00:52:15 dist-test-slave-05m0 krb5kdc[18004](info): commencing operation
krb5kdc: starting...
W20260925 00:52:16.309197 997 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:16 dist-test-slave-05m0 krb5kdc[18004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:16.431070 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:37811
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:37811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:16.538527 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:16.538826 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:16.538906 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:16.542963 18017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:16.543043 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:16.543081 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:16.543161 18017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:16.543200 18017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:16.548305 18017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:37811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:16.549304 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:16.550312 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:16.558542 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:16.558529 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:16.558529 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:16.558661 18017 server_base.cc:1034] running on GCE node
I20260925 00:52:16.559437 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:16.560527 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:16.561710 18017 hybrid_clock.cc:648] HybridClock initialized: now 1790297536561696 us; error 28 us; skew 500 ppm
Sep 25 00:52:16 dist-test-slave-05m0 krb5kdc[18004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:16.564435 18017 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:16.565312 18017 webserver.cc:460] Webserver started at http://127.0.249.126:39483/ using document root <none> and password file <none>
I20260925 00:52:16.565891 18017 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:16.565991 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:16.566224 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:16.568186 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "baab4c9a8c8949eca81033843693d075"
format_stamp: "Formatted at 2026-09-25 00:52:16 on dist-test-slave-05m0"
server_key: "f42dc2dad8a20d3c47510c48fa6d87f9"
server_key_iv: "d88a87ef63463195a164088821894471"
server_key_version: "encryptionkey@0"
I20260925 00:52:16.568758 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "baab4c9a8c8949eca81033843693d075"
format_stamp: "Formatted at 2026-09-25 00:52:16 on dist-test-slave-05m0"
server_key: "f42dc2dad8a20d3c47510c48fa6d87f9"
server_key_iv: "d88a87ef63463195a164088821894471"
server_key_version: "encryptionkey@0"
I20260925 00:52:16.572487 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:52:16.574898 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:16.576172 18017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:52:16.576308 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "baab4c9a8c8949eca81033843693d075"
format_stamp: "Formatted at 2026-09-25 00:52:16 on dist-test-slave-05m0"
server_key: "f42dc2dad8a20d3c47510c48fa6d87f9"
server_key_iv: "d88a87ef63463195a164088821894471"
server_key_version: "encryptionkey@0"
I20260925 00:52:16.576419 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:16.590842 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:16.593712 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:16.593911 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:16.600962 18017 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:37811
I20260925 00:52:16.600979 18084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:37811 every 8 connection(s)
I20260925 00:52:16.601989 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:16.604702 18085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:16.606915 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 18017
I20260925 00:52:16.607048 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:16.607375 997 external_mini_cluster.cc:1428] Setting key de07e8f0f28827166d7b2662d047add3
I20260925 00:52:16.610683 18085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075: Bootstrap starting.
I20260925 00:52:16.612739 18085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:16.613497 18085 log.cc:824] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:16.615881 18085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075: No bootstrap required, opened a new log
Sep 25 00:52:16 dist-test-slave-05m0 krb5kdc[18004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:16.619119 18085 raft_consensus.cc:348] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baab4c9a8c8949eca81033843693d075" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37811 } }
I20260925 00:52:16.619374 18085 raft_consensus.cc:374] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:16.619477 18085 raft_consensus.cc:729] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baab4c9a8c8949eca81033843693d075, State: Initialized, Role: FOLLOWER
I20260925 00:52:16.620031 18085 consensus_queue.cc:260] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baab4c9a8c8949eca81033843693d075" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37811 } }
I20260925 00:52:16.620164 18085 raft_consensus.cc:388] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:16.620280 18085 raft_consensus.cc:482] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:16.620394 18085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:16.621325 18085 raft_consensus.cc:504] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baab4c9a8c8949eca81033843693d075" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37811 } }
I20260925 00:52:16.621711 18085 leader_election.cc:302] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baab4c9a8c8949eca81033843693d075; no voters:
I20260925 00:52:16.622061 18085 leader_election.cc:288] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:16.622155 18090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:16.622385 18090 raft_consensus.cc:686] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [term 1 LEADER]: Becoming Leader. State: Replica: baab4c9a8c8949eca81033843693d075, State: Running, Role: LEADER
I20260925 00:52:16.622480 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:16.608787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.0.249.126:37811)
0925 00:52:16.609203 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:16.609218 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:16.609253 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:52:16.609929 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:16.609946 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:16.610257 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:16.610591 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:16.611391 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:16.612372 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:16.613294 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:16.613917 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:16.614089 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:16.617017 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:16.617022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:16.617033 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:16.617058 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:16.619316 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:16.619824 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:16.619829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:16.619833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:16.619940 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:16.620240 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:16.620246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:16.620249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:16.620719 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:16.620916 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:16.621221 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 00:52:16.621538 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260925 00:52:16.622965 18090 consensus_queue.cc:237] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baab4c9a8c8949eca81033843693d075" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37811 } }
I20260925 00:52:16.623332 18085 sys_catalog.cc:564] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:16.624172 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baab4c9a8c8949eca81033843693d075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baab4c9a8c8949eca81033843693d075" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37811 } } }
I20260925 00:52:16.624372 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:16.624694 18091 sys_catalog.cc:455] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baab4c9a8c8949eca81033843693d075. Latest consensus state: current_term: 1 leader_uuid: "baab4c9a8c8949eca81033843693d075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baab4c9a8c8949eca81033843693d075" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37811 } } }
I20260925 00:52:16.624889 18091 sys_catalog.cc:458] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:16.624971 18099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:16.627820 18099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:16.633949 18099 catalog_manager.cc:1347] Generated new cluster ID: b5eab7e5935d4447b76053802ed8ef64
I20260925 00:52:16.634032 18099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:16.650189 18099 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:16.651140 18099 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:16.657075 18099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075: Generated new TSK 0
I20260925 00:52:16.657822 18099 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:52:16.686584 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:37811
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:16.797015 18111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:16.797273 18111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:16.797313 18111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:16.801240 18111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:16.801306 18111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:16.801362 18111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:52:16.806466 18111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:37811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:16.807400 18111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:16.808332 18111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:16.817622 18117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:16.817660 18119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:16.817696 18116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:16.817822 18111 server_base.cc:1034] running on GCE node
I20260925 00:52:16.818311 18111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:16.818948 18111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:16.820147 18111 hybrid_clock.cc:648] HybridClock initialized: now 1790297536820071 us; error 43 us; skew 500 ppm
Sep 25 00:52:16 dist-test-slave-05m0 krb5kdc[18004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:16.822841 18111 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:52:16.823655 18111 webserver.cc:460] Webserver started at http://127.0.249.65:35531/ using document root <none> and password file <none>
I20260925 00:52:16.824313 18111 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:16.824375 18111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:16.824532 18111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:16.826542 18111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "f1822c86303e45a8b6b83f770b870f0d"
format_stamp: "Formatted at 2026-09-25 00:52:16 on dist-test-slave-05m0"
server_key: "1c3825451b164b7ffd3dfb05bc3361f0"
server_key_iv: "d949b195bcc02d81544aa3c2773e7982"
server_key_version: "encryptionkey@0"
I20260925 00:52:16.827054 18111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "f1822c86303e45a8b6b83f770b870f0d"
format_stamp: "Formatted at 2026-09-25 00:52:16 on dist-test-slave-05m0"
server_key: "1c3825451b164b7ffd3dfb05bc3361f0"
server_key_iv: "d949b195bcc02d81544aa3c2773e7982"
server_key_version: "encryptionkey@0"
I20260925 00:52:16.830503 18111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:52:16.832778 18126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:16.834012 18111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:16.834136 18111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "f1822c86303e45a8b6b83f770b870f0d"
format_stamp: "Formatted at 2026-09-25 00:52:16 on dist-test-slave-05m0"
server_key: "1c3825451b164b7ffd3dfb05bc3361f0"
server_key_iv: "d949b195bcc02d81544aa3c2773e7982"
server_key_version: "encryptionkey@0"
I20260925 00:52:16.834236 18111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:16.851672 18111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:16.854559 18111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:16.854758 18111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:16.855330 18111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:16.856139 18111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:16.856185 18111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:16.856252 18111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:16.856284 18111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:16.865077 18111 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:45387
I20260925 00:52:16.865094 18239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:45387 every 8 connection(s)
I20260925 00:52:16.865995 18111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:52:16.872363 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 18111
I20260925 00:52:16.872491 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:52:16.872792 997 external_mini_cluster.cc:1428] Setting key 36120f6f313c6155d717d12f96194bda
Sep 25 00:52:16 dist-test-slave-05m0 krb5kdc[18004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:16.880201 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:16.867765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:34383 (local address 127.0.249.126:37811)
0925 00:52:16.867897 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:16.867900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:16.868750 (+ 850us) server_negotiation.cc:407] Connection header received
0925 00:52:16.870089 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:16.870092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:16.870137 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:16.870212 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:16.871398 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:16.871917 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:16.873501 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:16.873756 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:16.873841 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:16.876607 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:16.876610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:16.876612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:16.876633 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:16.878082 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:16.878605 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:16.878610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:16.878611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:16.878693 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:16.879030 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:16.879035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:16.879040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:16.879317 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:16.879420 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:16.879916 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 00:52:16.880036 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:52:16.881042 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:16.868175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:37811 (local address 127.0.249.65:34383)
0925 00:52:16.868588 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:16.868629 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:16.869896 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:16.870365 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:16.870372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:16.870715 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:16.871255 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:16.871280 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:16.872018 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:16.872021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:16.873326 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:16.873359 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:16.873904 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:16.873908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:16.874093 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:16.874712 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:16.874745 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:16.876419 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:16.878209 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:16.878215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:16.878229 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:16.878499 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:16.878788 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:16.878792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:16.878794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:16.878935 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:16.879430 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:16.879439 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:16.879667 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 00:52:16.879852 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 00:52:16.880170 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:52:16.882318 18240 heartbeater.cc:347] Connected to a master server at 127.0.249.126:37811
I20260925 00:52:16.882534 18240 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:16.882918 18240 heartbeater.cc:511] Master 127.0.249.126:37811 requested a full tablet report, sending...
I20260925 00:52:16.884372 18049 ts_manager.cc:194] Registered new tserver with Master: f1822c86303e45a8b6b83f770b870f0d (127.0.249.65:45387)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
I20260925 00:52:16.885680 18049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:34383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:52:16.900712 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:37811
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:17.008647 18245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:17.008891 18245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:17.008987 18245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:17.012933 18245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:17.012987 18245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:17.013080 18245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:52:17.018311 18245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:37811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:17.019328 18245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:17.020321 18245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:17.030045 18250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:17.030066 18251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:17.030066 18253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:17.030119 18245 server_base.cc:1034] running on GCE node
I20260925 00:52:17.030709 18245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:17.031344 18245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:17.032528 18245 hybrid_clock.cc:648] HybridClock initialized: now 1790297537032514 us; error 28 us; skew 500 ppm
Sep 25 00:52:17 dist-test-slave-05m0 krb5kdc[18004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:17.035264 18245 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:52:17.036098 18245 webserver.cc:460] Webserver started at http://127.0.249.66:34613/ using document root <none> and password file <none>
I20260925 00:52:17.036672 18245 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:17.036761 18245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:17.037009 18245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:17.039113 18245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "b806c18adb6e4959b89a99901b2faf4b"
format_stamp: "Formatted at 2026-09-25 00:52:17 on dist-test-slave-05m0"
server_key: "97898a23c33bb32d91c71a9f1173673c"
server_key_iv: "e41ab0cd921ee6b6b4b94f8fc139486d"
server_key_version: "encryptionkey@0"
I20260925 00:52:17.039685 18245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "b806c18adb6e4959b89a99901b2faf4b"
format_stamp: "Formatted at 2026-09-25 00:52:17 on dist-test-slave-05m0"
server_key: "97898a23c33bb32d91c71a9f1173673c"
server_key_iv: "e41ab0cd921ee6b6b4b94f8fc139486d"
server_key_version: "encryptionkey@0"
I20260925 00:52:17.043394 18245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 00:52:17.045933 18260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:17.047197 18245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:17.047329 18245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "b806c18adb6e4959b89a99901b2faf4b"
format_stamp: "Formatted at 2026-09-25 00:52:17 on dist-test-slave-05m0"
server_key: "97898a23c33bb32d91c71a9f1173673c"
server_key_iv: "e41ab0cd921ee6b6b4b94f8fc139486d"
server_key_version: "encryptionkey@0"
I20260925 00:52:17.047446 18245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:17.066207 18245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:17.069276 18245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:17.069547 18245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:17.070189 18245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:17.071085 18245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:17.071158 18245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:17.071237 18245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:17.071297 18245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:17.080922 18245 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:37473
I20260925 00:52:17.080960 18373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:37473 every 8 connection(s)
I20260925 00:52:17.081924 18245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:52:17.086910 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 18245
I20260925 00:52:17.087040 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:52:17.087344 997 external_mini_cluster.cc:1428] Setting key bda3a009e9119907bbed30b53b594d16
Sep 25 00:52:17 dist-test-slave-05m0 krb5kdc[18004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:17.097074 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.083736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:47229 (local address 127.0.249.126:37811)
0925 00:52:17.083901 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.083906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.084753 (+ 847us) server_negotiation.cc:407] Connection header received
0925 00:52:17.085987 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.085989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.086035 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.086122 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:17.087555 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.088191 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.089804 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.090027 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.090125 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.092788 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:17.092795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:17.092801 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:17.092835 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:17.094689 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:17.095175 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:17.095181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:17.095183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:17.095276 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:17.095605 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:17.095611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:17.095614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:17.095869 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:17.095979 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.096664 (+ 685us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.096834 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 00:52:17.097541 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.084141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:37811 (local address 127.0.249.66:47229)
0925 00:52:17.084605 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:17.084648 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:17.085798 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:17.086284 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:17.086290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:17.086618 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:17.087324 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.087344 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.088315 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.088319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.089680 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.089695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.090203 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.090210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.090452 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.091166 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:17.091182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:17.092611 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:17.094807 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:17.094811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:17.094823 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:17.095072 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:17.095376 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:17.095378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:17.095379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:17.095504 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:17.095985 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:17.095990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:17.096266 (+ 276us) client_negotiation.cc:769] Sending connection context
0925 00:52:17.096420 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 00:52:17.096675 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:52:17.098830 18374 heartbeater.cc:347] Connected to a master server at 127.0.249.126:37811
I20260925 00:52:17.099118 18374 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:17.099716 18374 heartbeater.cc:511] Master 127.0.249.126:37811 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
I20260925 00:52:17.100955 18049 ts_manager.cc:194] Registered new tserver with Master: b806c18adb6e4959b89a99901b2faf4b (127.0.249.66:37473)
I20260925 00:52:17.101814 18049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:47229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:52:17.116482 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:37811
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:17.225666 18379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:17.225953 18379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:17.226073 18379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:17.230079 18379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:17.230170 18379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:17.230268 18379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:52:17.235491 18379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:37811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:17.236529 18379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:17.237663 18379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:17.247504 18384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:17.247512 18387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:17.247512 18385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:17.248222 18379 server_base.cc:1034] running on GCE node
I20260925 00:52:17.248607 18379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:17.249225 18379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:17.250366 18379 hybrid_clock.cc:648] HybridClock initialized: now 1790297537250355 us; error 34 us; skew 500 ppm
Sep 25 00:52:17 dist-test-slave-05m0 krb5kdc[18004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:17.253258 18379 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:52:17.254148 18379 webserver.cc:460] Webserver started at http://127.0.249.67:36375/ using document root <none> and password file <none>
I20260925 00:52:17.254706 18379 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:17.254796 18379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:17.255034 18379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:17.256990 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "b6d67735827e4852bde82b62c1ad793e"
format_stamp: "Formatted at 2026-09-25 00:52:17 on dist-test-slave-05m0"
server_key: "6f198fae48354385ee0b1e207ff50d54"
server_key_iv: "33cb0ca44b440c81dd2aaf4c42a8118a"
server_key_version: "encryptionkey@0"
I20260925 00:52:17.257592 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "b6d67735827e4852bde82b62c1ad793e"
format_stamp: "Formatted at 2026-09-25 00:52:17 on dist-test-slave-05m0"
server_key: "6f198fae48354385ee0b1e207ff50d54"
server_key_iv: "33cb0ca44b440c81dd2aaf4c42a8118a"
server_key_version: "encryptionkey@0"
I20260925 00:52:17.261062 18379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 00:52:17.263607 18394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:17.264811 18379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:17.264945 18379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "b6d67735827e4852bde82b62c1ad793e"
format_stamp: "Formatted at 2026-09-25 00:52:17 on dist-test-slave-05m0"
server_key: "6f198fae48354385ee0b1e207ff50d54"
server_key_iv: "33cb0ca44b440c81dd2aaf4c42a8118a"
server_key_version: "encryptionkey@0"
I20260925 00:52:17.265051 18379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:17.274974 18379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:17.277638 18379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:17.277835 18379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:17.278388 18379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:17.279194 18379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:17.279270 18379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:17.279333 18379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:17.279382 18379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:17.287744 18379 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:42437
I20260925 00:52:17.287771 18507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:42437 every 8 connection(s)
I20260925 00:52:17.288591 18379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:52:17.292078 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 18379
I20260925 00:52:17.292196 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:52:17.292532 997 external_mini_cluster.cc:1428] Setting key 4533a584621f69afc421340a55df277e
Sep 25 00:52:17 dist-test-slave-05m0 krb5kdc[18004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:17.301318 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.290387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:33775 (local address 127.0.249.126:37811)
0925 00:52:17.290530 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.290534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.291368 (+ 834us) server_negotiation.cc:407] Connection header received
0925 00:52:17.292399 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.292402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.292447 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.292511 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:17.293909 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.294504 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.295626 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.295837 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.295914 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.298276 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:17.298279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:17.298281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:17.298304 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:17.299596 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:17.300066 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:17.300069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:17.300070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:17.300145 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:17.300431 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:17.300433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:17.300434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:17.300629 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:17.300703 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.301117 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.301205 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 00:52:17.302124 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.290804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:37811 (local address 127.0.249.67:33775)
0925 00:52:17.291219 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:17.291258 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:17.292127 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:17.292716 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:17.292725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:17.293108 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:17.293786 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.293802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.294658 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.294661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.295496 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.295505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.295960 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.295962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.296096 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.296625 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:17.296646 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:17.298122 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:17.299711 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:17.299715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:17.299726 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:17.299965 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:17.300228 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:17.300230 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:17.300231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:17.300355 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:17.300714 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:17.300718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:17.300925 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 00:52:17.301074 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 00:52:17.301316 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
I20260925 00:52:17.303165 18508 heartbeater.cc:347] Connected to a master server at 127.0.249.126:37811
I20260925 00:52:17.303395 18508 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:17.303850 18508 heartbeater.cc:511] Master 127.0.249.126:37811 requested a full tablet report, sending...
I20260925 00:52:17.304863 18049 ts_manager.cc:194] Registered new tserver with Master: b6d67735827e4852bde82b62c1ad793e (127.0.249.67:42437)
I20260925 00:52:17.305613 18049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:33775
I20260925 00:52:17.306449 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:52:17.314921 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.308342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.0.249.126:37811)
0925 00:52:17.308448 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.308451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.308632 (+ 181us) server_negotiation.cc:407] Connection header received
0925 00:52:17.308727 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.308729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.308763 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.308821 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:17.309464 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.309943 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.310718 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.310914 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.310986 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.312268 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:17.312270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:17.312272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:17.312290 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:17.313586 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:17.313954 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:17.313956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:17.313957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:17.314026 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:17.314252 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:17.314254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:17.314255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:17.314438 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:17.314540 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.314732 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.314815 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:52:17.317786 18048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:52:17.320694 18048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:52:17.340123 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.333784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45387 (local address 127.0.0.1:60000)
0925 00:52:17.334337 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:17.334393 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:17.334520 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:17.335504 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:17.335510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:17.335532 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:17.335681 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.335688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.337246 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.337249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.338639 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.338648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.339717 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.339720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.339811 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.339834 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:52:17.339882 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 00:52:17.339945 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":68,"thread_start_us":67,"threads_started":1}
I20260925 00:52:17.340453 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.334598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.0.249.65:45387)
0925 00:52:17.335008 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.335023 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.335046 (+ 23us) server_negotiation.cc:407] Connection header received
0925 00:52:17.335122 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.335128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.335330 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.335830 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:17.335970 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.337097 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.338774 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.339560 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.340037 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.340180 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.340232 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.340316 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":117,"threads_started":1}
I20260925 00:52:17.340617 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.334800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.0.249.67:42437)
0925 00:52:17.335162 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.335173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.335196 (+ 23us) server_negotiation.cc:407] Connection header received
0925 00:52:17.335272 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.335277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.335482 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.335633 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:17.336073 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.337011 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.338626 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.339266 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.339393 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.340301 (+ 908us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.340354 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.340446 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":131,"threads_started":1}
I20260925 00:52:17.339956 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.334050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42437 (local address 127.0.0.1:36882)
0925 00:52:17.334650 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:17.334665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:17.334759 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:17.335825 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:17.335829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:17.335852 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:17.335977 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.335983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.337176 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.337180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.338501 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.338511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.339391 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.339395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.339519 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.339559 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 00:52:17.339661 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 00:52:17.339793 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":58,"spinlock_wait_cycles":5376,"thread_start_us":95,"threads_started":1}
I20260925 00:52:17.342314 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.334239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:37473 (local address 127.0.0.1:54972)
0925 00:52:17.335812 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:17.335825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:17.335906 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:17.338253 (+ 2347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:17.338256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:17.338272 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:17.338364 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.338370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.339736 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.339739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.340840 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.340851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.341924 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.341928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.342022 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.342043 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:52:17.342123 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 00:52:17.342188 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1507,"spinlock_wait_cycles":2176,"thread_start_us":107,"threads_started":1}
I20260925 00:52:17.342859 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.334762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.0.249.66:37473)
0925 00:52:17.337611 (+ 2849us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.337622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.337689 (+ 67us) server_negotiation.cc:407] Connection header received
0925 00:52:17.337763 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.337768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.337947 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.338102 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:17.338483 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.339603 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.340983 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.341800 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.341925 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.342053 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.342648 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.342740 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2745,"thread_start_us":108,"threads_started":1}
I20260925 00:52:17.343734 18442 tablet_service.cc:1467] Processing CreateTablet for tablet 691c62c5adb24a048734fc9f2a231504 (DEFAULT_TABLE table=test-table [id=591e35f364a34482b04a3daa25b9314b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:17.343737 18174 tablet_service.cc:1467] Processing CreateTablet for tablet 691c62c5adb24a048734fc9f2a231504 (DEFAULT_TABLE table=test-table [id=591e35f364a34482b04a3daa25b9314b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:17.344893 18442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691c62c5adb24a048734fc9f2a231504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:17.344893 18174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691c62c5adb24a048734fc9f2a231504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:17.352293 18526 tablet_bootstrap.cc:492] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d: Bootstrap starting.
I20260925 00:52:17.352723 18525 tablet_bootstrap.cc:492] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: Bootstrap starting.
I20260925 00:52:17.353765 18308 tablet_service.cc:1467] Processing CreateTablet for tablet 691c62c5adb24a048734fc9f2a231504 (DEFAULT_TABLE table=test-table [id=591e35f364a34482b04a3daa25b9314b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:17.354843 18308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691c62c5adb24a048734fc9f2a231504. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:17.355362 18526 tablet_bootstrap.cc:654] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:17.355440 18525 tablet_bootstrap.cc:654] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:17.356433 18526 log.cc:824] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:17.356581 18525 log.cc:824] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:17.358215 18526 tablet_bootstrap.cc:492] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d: No bootstrap required, opened a new log
I20260925 00:52:17.358423 18526 ts_tablet_manager.cc:1397] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:52:17.359727 18525 tablet_bootstrap.cc:492] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: No bootstrap required, opened a new log
I20260925 00:52:17.359884 18525 ts_tablet_manager.cc:1397] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 00:52:17.361953 18529 tablet_bootstrap.cc:492] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: Bootstrap starting.
I20260925 00:52:17.362534 18526 raft_consensus.cc:348] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } }
I20260925 00:52:17.362795 18526 raft_consensus.cc:374] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:17.362849 18526 raft_consensus.cc:729] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1822c86303e45a8b6b83f770b870f0d, State: Initialized, Role: FOLLOWER
I20260925 00:52:17.363394 18526 consensus_queue.cc:260] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } }
I20260925 00:52:17.364420 18526 ts_tablet_manager.cc:1428] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:52:17.364794 18240 heartbeater.cc:503] Master 127.0.249.126:37811 was elected leader, sending a full tablet report...
I20260925 00:52:17.365569 18529 tablet_bootstrap.cc:654] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:17.365908 18525 raft_consensus.cc:348] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } }
I20260925 00:52:17.366180 18525 raft_consensus.cc:374] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:17.366237 18525 raft_consensus.cc:729] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d67735827e4852bde82b62c1ad793e, State: Initialized, Role: FOLLOWER
I20260925 00:52:17.366475 18529 log.cc:824] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:17.366727 18525 consensus_queue.cc:260] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } }
W20260925 00:52:17.367211 18241 tablet.cc:2369] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:17.367398 18508 heartbeater.cc:503] Master 127.0.249.126:37811 was elected leader, sending a full tablet report...
I20260925 00:52:17.368455 18525 ts_tablet_manager.cc:1428] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260925 00:52:17.370491 18529 tablet_bootstrap.cc:492] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: No bootstrap required, opened a new log
I20260925 00:52:17.370721 18529 ts_tablet_manager.cc:1397] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 00:52:17.373715 18529 raft_consensus.cc:348] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } }
I20260925 00:52:17.373922 18529 raft_consensus.cc:374] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:17.373960 18529 raft_consensus.cc:729] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b806c18adb6e4959b89a99901b2faf4b, State: Initialized, Role: FOLLOWER
I20260925 00:52:17.374289 18529 consensus_queue.cc:260] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } }
I20260925 00:52:17.374833 18374 heartbeater.cc:503] Master 127.0.249.126:37811 was elected leader, sending a full tablet report...
I20260925 00:52:17.375140 18529 ts_tablet_manager.cc:1428] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
W20260925 00:52:17.539541 18509 tablet.cc:2369] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:17.576985 18533 raft_consensus.cc:482] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:17.577200 18533 raft_consensus.cc:504] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } }
I20260925 00:52:17.578428 18533 leader_election.cc:288] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1822c86303e45a8b6b83f770b870f0d (127.0.249.65:45387), b6d67735827e4852bde82b62c1ad793e (127.0.249.67:42437)
I20260925 00:52:17.582870 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.578651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45387 (local address 127.0.249.66:42395)
0925 00:52:17.578812 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:17.578829 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:17.578923 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:17.579321 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:17.579342 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:17.579368 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:17.579491 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.579496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.580358 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.580362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.581700 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.581711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.582467 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.582470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.582547 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.582567 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:52:17.582630 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 00:52:17.582722 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":66}
I20260925 00:52:17.582908 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.578854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:42395 (local address 127.0.249.65:45387)
0925 00:52:17.579097 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.579102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.579116 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:52:17.579174 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.579180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.579233 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.579310 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:17.579616 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.580248 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.581817 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.582357 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.582475 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.582526 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.582684 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.582761 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
W20260925 00:52:17.583074 18375 tablet.cc:2369] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:17.583220 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.578662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:42437 (local address 127.0.249.66:54221)
0925 00:52:17.579134 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:17.579147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:17.579238 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:17.579778 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:17.579781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:17.579800 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:17.579906 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.579912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.580672 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.580676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.581924 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:17.581931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.582729 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.582732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:17.582820 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.582841 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 00:52:17.582913 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:52:17.582989 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":191,"thread_start_us":101,"threads_started":1}
I20260925 00:52:17.583680 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.578997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:54221 (local address 127.0.249.67:42437)
0925 00:52:17.579132 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.579135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.579228 (+ 93us) server_negotiation.cc:407] Connection header received
0925 00:52:17.579623 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.579625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.579673 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.579730 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:17.579998 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.580561 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.582025 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.582601 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.582731 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.582775 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.583479 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.583566 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:52:17.583669 18194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691c62c5adb24a048734fc9f2a231504" candidate_uuid: "b806c18adb6e4959b89a99901b2faf4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1822c86303e45a8b6b83f770b870f0d" is_pre_election: true
I20260925 00:52:17.584025 18194 raft_consensus.cc:2454] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b806c18adb6e4959b89a99901b2faf4b in term 0.
I20260925 00:52:17.584237 18462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691c62c5adb24a048734fc9f2a231504" candidate_uuid: "b806c18adb6e4959b89a99901b2faf4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d67735827e4852bde82b62c1ad793e" is_pre_election: true
I20260925 00:52:17.584525 18462 raft_consensus.cc:2454] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b806c18adb6e4959b89a99901b2faf4b in term 0.
I20260925 00:52:17.584578 18263 leader_election.cc:302] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b806c18adb6e4959b89a99901b2faf4b, f1822c86303e45a8b6b83f770b870f0d; no voters:
I20260925 00:52:17.584883 18533 raft_consensus.cc:2781] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:52:17.585004 18533 raft_consensus.cc:482] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:52:17.585081 18533 raft_consensus.cc:3037] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:17.586230 18533 raft_consensus.cc:504] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } }
I20260925 00:52:17.586580 18533 leader_election.cc:288] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [CANDIDATE]: Term 1 election: Requested vote from peers f1822c86303e45a8b6b83f770b870f0d (127.0.249.65:45387), b6d67735827e4852bde82b62c1ad793e (127.0.249.67:42437)
I20260925 00:52:17.586953 18194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691c62c5adb24a048734fc9f2a231504" candidate_uuid: "b806c18adb6e4959b89a99901b2faf4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1822c86303e45a8b6b83f770b870f0d"
I20260925 00:52:17.586975 18462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691c62c5adb24a048734fc9f2a231504" candidate_uuid: "b806c18adb6e4959b89a99901b2faf4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d67735827e4852bde82b62c1ad793e"
I20260925 00:52:17.587090 18194 raft_consensus.cc:3037] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:17.587113 18462 raft_consensus.cc:3037] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:17.588475 18194 raft_consensus.cc:2454] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b806c18adb6e4959b89a99901b2faf4b in term 1.
I20260925 00:52:17.588479 18462 raft_consensus.cc:2454] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b806c18adb6e4959b89a99901b2faf4b in term 1.
I20260925 00:52:17.588820 18264 leader_election.cc:302] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b6d67735827e4852bde82b62c1ad793e, b806c18adb6e4959b89a99901b2faf4b, f1822c86303e45a8b6b83f770b870f0d; no voters:
I20260925 00:52:17.589042 18533 raft_consensus.cc:2781] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:17.589205 18533 raft_consensus.cc:686] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 1 LEADER]: Becoming Leader. State: Replica: b806c18adb6e4959b89a99901b2faf4b, State: Running, Role: LEADER
I20260925 00:52:17.589645 18533 consensus_queue.cc:237] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } }
I20260925 00:52:17.592732 18047 catalog_manager.cc:5495] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b reported cstate change: term changed from 0 to 1, leader changed from <none> to b806c18adb6e4959b89a99901b2faf4b (127.0.249.66). New cstate: current_term: 1 leader_uuid: "b806c18adb6e4959b89a99901b2faf4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b806c18adb6e4959b89a99901b2faf4b" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 37473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20260925 00:52:17.643261 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.640118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.0.249.66:37473)
0925 00:52:17.640266 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.640269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.640281 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:52:17.640353 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.640355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.640408 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.640496 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:17.640716 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.641199 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.642028 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.642240 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.642317 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.642560 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:17.642954 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:17.643041 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.643095 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.643141 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 00:52:17.648917 18462 raft_consensus.cc:1261] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [term 1 FOLLOWER]: Refusing update from remote peer b806c18adb6e4959b89a99901b2faf4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:17.648917 18194 raft_consensus.cc:1261] T 691c62c5adb24a048734fc9f2a231504 P f1822c86303e45a8b6b83f770b870f0d [term 1 FOLLOWER]: Refusing update from remote peer b806c18adb6e4959b89a99901b2faf4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:17.649653 18535 consensus_queue.cc:1035] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1822c86303e45a8b6b83f770b870f0d" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:17.649830 18533 consensus_queue.cc:1035] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d67735827e4852bde82b62c1ad793e" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:17.656922 18538 mvcc.cc:205] Tried to move back new op lower bound from 7333058714203123712 to 7333058713970085888. Current Snapshot: MvccSnapshot[applied={T|T < 7333058714203123712}]
I20260925 00:52:17.657294 18539 mvcc.cc:205] Tried to move back new op lower bound from 7333058714203123712 to 7333058713970085888. Current Snapshot: MvccSnapshot[applied={T|T < 7333058714203123712}]
I20260925 00:52:17.659679 18540 mvcc.cc:205] Tried to move back new op lower bound from 7333058714203123712 to 7333058713970085888. Current Snapshot: MvccSnapshot[applied={T|T < 7333058714203123712}]
I20260925 00:52:17.665994 18047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:17.666167 18047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:17.668833 18047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P baab4c9a8c8949eca81033843693d075: Sending DeleteTablet for 3 replicas of tablet 691c62c5adb24a048734fc9f2a231504
I20260925 00:52:17.669776 18174 tablet_service.cc:1514] Processing DeleteTablet for tablet 691c62c5adb24a048734fc9f2a231504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:17 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:60000
I20260925 00:52:17.669838 18308 tablet_service.cc:1514] Processing DeleteTablet for tablet 691c62c5adb24a048734fc9f2a231504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:17 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:54972
I20260925 00:52:17.669837 18442 tablet_service.cc:1514] Processing DeleteTablet for tablet 691c62c5adb24a048734fc9f2a231504 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:17 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:36882
I20260925 00:52:17.670512 18548 tablet_replica.cc:330] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: stopping tablet replica
I20260925 00:52:17.670542 18549 tablet_replica.cc:330] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: stopping tablet replica
I20260925 00:52:17.670889 18549 raft_consensus.cc:2229] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:52:17.671250 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 18111
I20260925 00:52:17.671283 18549 raft_consensus.cc:2258] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:17.673628 18549 ts_tablet_manager.cc:1905] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:17.676692 18548 raft_consensus.cc:2229] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:52:17.677165 18548 raft_consensus.cc:2258] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:17.677196 18549 ts_tablet_manager.cc:1918] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:17.677305 18549 log.cc:1196] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/691c62c5adb24a048734fc9f2a231504
I20260925 00:52:17.677786 18549 ts_tablet_manager.cc:1939] T 691c62c5adb24a048734fc9f2a231504 P b6d67735827e4852bde82b62c1ad793e: Deleting consensus metadata
I20260925 00:52:17.679170 18036 catalog_manager.cc:4841] TS b6d67735827e4852bde82b62c1ad793e (127.0.249.67:42437): tablet 691c62c5adb24a048734fc9f2a231504 (table test-table [id=591e35f364a34482b04a3daa25b9314b]) successfully deleted
W20260925 00:52:17.682730 18035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:45387 (error 108)
I20260925 00:52:17.683751 18548 ts_tablet_manager.cc:1905] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:17.683877 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.683506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45387 (local address 127.0.0.1:60008)
0925 00:52:17.683692 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:17.683772 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260925 00:52:17.684135 18035 catalog_manager.cc:4584] TS f1822c86303e45a8b6b83f770b870f0d (127.0.249.65:45387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 691c62c5adb24a048734fc9f2a231504: Network error: Client connection negotiation failed: client connection to 127.0.249.65:45387: connect: Connection refused (error 111)
I20260925 00:52:17.686318 18548 ts_tablet_manager.cc:1918] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:17.686453 18548 log.cc:1196] T 691c62c5adb24a048734fc9f2a231504 P b806c18adb6e4959b89a99901b2faf4b: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/691c62c5adb24a048734fc9f2a231504
I20260925 00:52:17.686625 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 18245
I20260925 00:52:17.692798 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 18379
I20260925 00:52:17.693238 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.692933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:37473 (local address 127.0.0.1:54990)
0925 00:52:17.693087 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:17.693149 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.66:37473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 00:52:17.698467 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 18017
2026-09-25T00:52:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2428 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T00:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:17Z Disabled control of system clock
I20260925 00:52:17.727911 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46377
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:43811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:52:17.835683 18554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:17.835914 18554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:17.835947 18554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:17.839804 18554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:17.839856 18554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:17.839872 18554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:17.839888 18554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:17.839902 18554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:17.844609 18554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:17.845559 18554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:17.846827 18554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:17.854894 18559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:17.854915 18560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:17.854974 18554 server_base.cc:1034] running on GCE node
W20260925 00:52:17.855223 18562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:17.855762 18554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:17.856750 18554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:17.857901 18554 hybrid_clock.cc:648] HybridClock initialized: now 1790297537857872 us; error 40 us; skew 500 ppm
I20260925 00:52:17.859551 18554 webserver.cc:460] Webserver started at http://127.0.249.126:44387/ using document root <none> and password file <none>
I20260925 00:52:17.860070 18554 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:17.860131 18554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:17.860277 18554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:17.862211 18554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "c8169ab534964f1899a8a9a7bb04ee76"
format_stamp: "Formatted at 2026-09-25 00:52:17 on dist-test-slave-05m0"
server_key: "b9c33976bcc3365af6931c073ba3f119"
server_key_iv: "96b0367ee1767a019691471364576f6e"
server_key_version: "encryptionkey@0"
I20260925 00:52:17.862706 18554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "c8169ab534964f1899a8a9a7bb04ee76"
format_stamp: "Formatted at 2026-09-25 00:52:17 on dist-test-slave-05m0"
server_key: "b9c33976bcc3365af6931c073ba3f119"
server_key_iv: "96b0367ee1767a019691471364576f6e"
server_key_version: "encryptionkey@0"
I20260925 00:52:17.866055 18554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:52:17.868251 18568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:17.869370 18554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:17.869547 18554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "c8169ab534964f1899a8a9a7bb04ee76"
format_stamp: "Formatted at 2026-09-25 00:52:17 on dist-test-slave-05m0"
server_key: "b9c33976bcc3365af6931c073ba3f119"
server_key_iv: "96b0367ee1767a019691471364576f6e"
server_key_version: "encryptionkey@0"
I20260925 00:52:17.869679 18554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:17.879091 18554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:17.879664 18554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:17.879853 18554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:17.886739 18554 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:46377
I20260925 00:52:17.886770 18620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:46377 every 8 connection(s)
I20260925 00:52:17.887580 18554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:17.890342 18621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:17.892906 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 18554
I20260925 00:52:17.893003 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:17.893288 997 external_mini_cluster.cc:1428] Setting key 93e9135c96e91c70dcb9362d1189db33
I20260925 00:52:17.895745 18621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76: Bootstrap starting.
I20260925 00:52:17.898041 18621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:17.898870 18621 log.cc:824] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:17.900908 18621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76: No bootstrap required, opened a new log
I20260925 00:52:17.902257 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:17.894612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.0.249.126:46377)
0925 00:52:17.895184 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:17.895198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:17.895231 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:52:17.895830 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:17.895845 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:17.896242 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:17.896498 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:17.896898 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.897890 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.899048 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:17.899655 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:17.899788 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:17.900600 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:17.900605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:17.900617 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:17.900963 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:17.901046 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:17.901082 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:17.901157 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:17.901324 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 00:52:17.901614 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":125,"threads_started":1}
I20260925 00:52:17.904742 18621 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8169ab534964f1899a8a9a7bb04ee76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46377 } }
I20260925 00:52:17.904963 18621 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:17.905040 18621 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8169ab534964f1899a8a9a7bb04ee76, State: Initialized, Role: FOLLOWER
I20260925 00:52:17.905484 18621 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8169ab534964f1899a8a9a7bb04ee76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46377 } }
I20260925 00:52:17.905629 18621 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:17.905718 18621 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:17.905813 18621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:17.906632 18621 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8169ab534964f1899a8a9a7bb04ee76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46377 } }
I20260925 00:52:17.906958 18621 leader_election.cc:302] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8169ab534964f1899a8a9a7bb04ee76; no voters:
I20260925 00:52:17.907284 18621 leader_election.cc:288] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:17.907344 18626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:17.907536 18626 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [term 1 LEADER]: Becoming Leader. State: Replica: c8169ab534964f1899a8a9a7bb04ee76, State: Running, Role: LEADER
I20260925 00:52:17.907812 18626 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8169ab534964f1899a8a9a7bb04ee76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46377 } }
I20260925 00:52:17.908470 18621 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:17.909512 18627 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8169ab534964f1899a8a9a7bb04ee76. Latest consensus state: current_term: 1 leader_uuid: "c8169ab534964f1899a8a9a7bb04ee76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8169ab534964f1899a8a9a7bb04ee76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46377 } } }
I20260925 00:52:17.909634 18627 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:17.909934 18628 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8169ab534964f1899a8a9a7bb04ee76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8169ab534964f1899a8a9a7bb04ee76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46377 } } }
I20260925 00:52:17.910000 18635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:17.910066 18628 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:17.913196 18635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:17.918607 18635 catalog_manager.cc:1347] Generated new cluster ID: 9977b13b45cd4b979ed3e557042de095
I20260925 00:52:17.918684 18635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:17.940408 18635 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:17.941614 18635 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:17.955966 18635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76: Generated new TSK 0
I20260925 00:52:17.956763 18635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:17.965621 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46377
--builtin_ntp_servers=127.0.249.84:43811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:52:18.075774 18645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:18.076030 18645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:18.076092 18645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:18.080084 18645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:18.080132 18645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:18.080214 18645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:52:18.084894 18645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:18.085970 18645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:18.087332 18645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:18.096662 18650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:18.096745 18651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:18.096662 18653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:18.096858 18645 server_base.cc:1034] running on GCE node
I20260925 00:52:18.097307 18645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:18.097963 18645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:18.099119 18645 hybrid_clock.cc:648] HybridClock initialized: now 1790297538099104 us; error 27 us; skew 500 ppm
I20260925 00:52:18.100865 18645 webserver.cc:460] Webserver started at http://127.0.249.65:42961/ using document root <none> and password file <none>
I20260925 00:52:18.101493 18645 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:18.101559 18645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:18.101785 18645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:18.103736 18645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "a5641c9be81846aaa5c393fb4b818386"
format_stamp: "Formatted at 2026-09-25 00:52:18 on dist-test-slave-05m0"
server_key: "8a164c66977905ef095af372b32a52b0"
server_key_iv: "74a07109c589440191d90095a16ab9aa"
server_key_version: "encryptionkey@0"
I20260925 00:52:18.104275 18645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "a5641c9be81846aaa5c393fb4b818386"
format_stamp: "Formatted at 2026-09-25 00:52:18 on dist-test-slave-05m0"
server_key: "8a164c66977905ef095af372b32a52b0"
server_key_iv: "74a07109c589440191d90095a16ab9aa"
server_key_version: "encryptionkey@0"
I20260925 00:52:18.107754 18645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:52:18.110217 18659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:18.111459 18645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:18.111589 18645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "a5641c9be81846aaa5c393fb4b818386"
format_stamp: "Formatted at 2026-09-25 00:52:18 on dist-test-slave-05m0"
server_key: "8a164c66977905ef095af372b32a52b0"
server_key_iv: "74a07109c589440191d90095a16ab9aa"
server_key_version: "encryptionkey@0"
I20260925 00:52:18.111698 18645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:18.130553 18645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:18.131362 18645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:18.131553 18645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:18.132128 18645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:18.133030 18645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:18.133082 18645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:18.133152 18645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:18.133184 18645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:18.141703 18645 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:35465
I20260925 00:52:18.141739 18772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:35465 every 8 connection(s)
I20260925 00:52:18.142575 18645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:52:18.151587 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.144347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:47139 (local address 127.0.249.126:46377)
0925 00:52:18.144484 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.144495 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.145569 (+ 1074us) server_negotiation.cc:407] Connection header received
0925 00:52:18.146994 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.146997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.147060 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.147143 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:18.148017 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.148550 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.149718 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.149948 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.150063 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.151091 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:18.151095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:18.151097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:18.151110 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:18.151138 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:18.151147 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:18.151224 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.151383 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.151445 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 00:52:18.152076 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 18645
I20260925 00:52:18.152204 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:52:18.152486 997 external_mini_cluster.cc:1428] Setting key a03c664cbd532fc52370d9589900789a
I20260925 00:52:18.152480 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.144737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46377 (local address 127.0.249.65:47139)
0925 00:52:18.145367 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:18.145465 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:18.146769 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:18.147312 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:18.147318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:18.147401 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:18.147866 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.147878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.148679 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.148684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.149582 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.149591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.150098 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.150101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.150269 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.150839 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:52:18.150840 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:18.150880 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:52:18.150939 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:18.151238 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:18.151251 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:18.151260 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 00:52:18.151382 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 00:52:18.151652 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260925 00:52:18.153733 18773 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46377
I20260925 00:52:18.153960 18773 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:18.154445 18773 heartbeater.cc:511] Master 127.0.249.126:46377 requested a full tablet report, sending...
I20260925 00:52:18.154631 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46377
--builtin_ntp_servers=127.0.249.84:43811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:52:18.156051 18584 ts_manager.cc:194] Registered new tserver with Master: a5641c9be81846aaa5c393fb4b818386 (127.0.249.65:35465)
I20260925 00:52:18.157462 18584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.65:47139
W20260925 00:52:18.264904 18776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:18.265192 18776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:18.265259 18776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:18.269304 18776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:18.269373 18776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:18.269476 18776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:52:18.274344 18776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:18.275434 18776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:18.276754 18776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:18.285897 18781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:18.285925 18782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:18.285955 18784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:18.286126 18776 server_base.cc:1034] running on GCE node
I20260925 00:52:18.286618 18776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:18.287263 18776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:18.288447 18776 hybrid_clock.cc:648] HybridClock initialized: now 1790297538288434 us; error 25 us; skew 500 ppm
I20260925 00:52:18.290271 18776 webserver.cc:460] Webserver started at http://127.0.249.66:38475/ using document root <none> and password file <none>
I20260925 00:52:18.290901 18776 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:18.290990 18776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:18.291211 18776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:18.293733 18776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "e9296451d6e941c6b9de1e31e4a19487"
format_stamp: "Formatted at 2026-09-25 00:52:18 on dist-test-slave-05m0"
server_key: "57e4b19f2ba53652fe253ef58ecf84b5"
server_key_iv: "f25d49e0fead8ffd77c25dfc88367d69"
server_key_version: "encryptionkey@0"
I20260925 00:52:18.294286 18776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "e9296451d6e941c6b9de1e31e4a19487"
format_stamp: "Formatted at 2026-09-25 00:52:18 on dist-test-slave-05m0"
server_key: "57e4b19f2ba53652fe253ef58ecf84b5"
server_key_iv: "f25d49e0fead8ffd77c25dfc88367d69"
server_key_version: "encryptionkey@0"
I20260925 00:52:18.297827 18776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:52:18.300258 18790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:18.301661 18776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:18.301790 18776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "e9296451d6e941c6b9de1e31e4a19487"
format_stamp: "Formatted at 2026-09-25 00:52:18 on dist-test-slave-05m0"
server_key: "57e4b19f2ba53652fe253ef58ecf84b5"
server_key_iv: "f25d49e0fead8ffd77c25dfc88367d69"
server_key_version: "encryptionkey@0"
I20260925 00:52:18.301905 18776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:18.315260 18776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:18.315980 18776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:18.316169 18776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:18.316722 18776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:18.317713 18776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:18.317783 18776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:18.317847 18776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:18.317889 18776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:18.327111 18776 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:34769
I20260925 00:52:18.327135 18903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:34769 every 8 connection(s)
I20260925 00:52:18.327970 18776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:52:18.331461 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 18776
I20260925 00:52:18.331599 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:52:18.331908 997 external_mini_cluster.cc:1428] Setting key 7dce9bb5018f1c78d40f14dfa4e5ae9f
I20260925 00:52:18.333957 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46377
--builtin_ntp_servers=127.0.249.84:43811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:52:18.337478 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.329715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:55667 (local address 127.0.249.126:46377)
0925 00:52:18.329848 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.329851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.330852 (+ 1001us) server_negotiation.cc:407] Connection header received
0925 00:52:18.332646 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.332650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.332696 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.332796 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:18.333808 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.334420 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.335531 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.335758 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.335873 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.336961 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:18.336964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:18.336967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:18.336985 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:18.337003 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:18.337011 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:18.337088 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.337254 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.337314 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:52:18.338327 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.330101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46377 (local address 127.0.249.66:55667)
0925 00:52:18.330715 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:18.330753 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:18.332408 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:18.332968 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:18.332974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:18.333056 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:18.333660 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.333675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.334545 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.334548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.335403 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.335412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.335905 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.335909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.336076 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.336702 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:52:18.336704 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:18.336743 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:52:18.336804 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:18.337102 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:18.337113 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:18.337121 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 00:52:18.337251 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 00:52:18.337534 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260925 00:52:18.339598 18904 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46377
I20260925 00:52:18.339836 18904 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:18.340317 18904 heartbeater.cc:511] Master 127.0.249.126:46377 requested a full tablet report, sending...
I20260925 00:52:18.341349 18584 ts_manager.cc:194] Registered new tserver with Master: e9296451d6e941c6b9de1e31e4a19487 (127.0.249.66:34769)
I20260925 00:52:18.341948 18584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.66:55667
W20260925 00:52:18.443521 18907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:18.443764 18907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:18.443801 18907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:18.447675 18907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:18.447723 18907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:18.447773 18907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:52:18.452473 18907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:18.453460 18907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:18.454758 18907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:18.464440 18913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:18.464392 18915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:18.464617 18907 server_base.cc:1034] running on GCE node
W20260925 00:52:18.464388 18912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:18.465032 18907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:18.465761 18907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:18.466923 18907 hybrid_clock.cc:648] HybridClock initialized: now 1790297538466907 us; error 27 us; skew 500 ppm
I20260925 00:52:18.468918 18907 webserver.cc:460] Webserver started at http://127.0.249.67:36637/ using document root <none> and password file <none>
I20260925 00:52:18.469545 18907 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:18.469611 18907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:18.469864 18907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:18.472074 18907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "0a728ba61d02468bb75bf90cb7a7254c"
format_stamp: "Formatted at 2026-09-25 00:52:18 on dist-test-slave-05m0"
server_key: "ce2f4342fdb9b7ae9295e94cfd7b0149"
server_key_iv: "59451c5ecc7a444bbaeb601932ba4336"
server_key_version: "encryptionkey@0"
I20260925 00:52:18.472734 18907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "0a728ba61d02468bb75bf90cb7a7254c"
format_stamp: "Formatted at 2026-09-25 00:52:18 on dist-test-slave-05m0"
server_key: "ce2f4342fdb9b7ae9295e94cfd7b0149"
server_key_iv: "59451c5ecc7a444bbaeb601932ba4336"
server_key_version: "encryptionkey@0"
I20260925 00:52:18.476936 18907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 00:52:18.479595 18921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:18.480895 18907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:18.481057 18907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "0a728ba61d02468bb75bf90cb7a7254c"
format_stamp: "Formatted at 2026-09-25 00:52:18 on dist-test-slave-05m0"
server_key: "ce2f4342fdb9b7ae9295e94cfd7b0149"
server_key_iv: "59451c5ecc7a444bbaeb601932ba4336"
server_key_version: "encryptionkey@0"
I20260925 00:52:18.481172 18907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:18.493749 18907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:18.494431 18907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:18.494665 18907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:18.495280 18907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:18.496271 18907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:18.496325 18907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:18.496407 18907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:18.496465 18907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:18.506213 18907 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:36833
I20260925 00:52:18.506253 19034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:36833 every 8 connection(s)
I20260925 00:52:18.507073 18907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:52:18.509943 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 18907
I20260925 00:52:18.510051 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:52:18.510352 997 external_mini_cluster.cc:1428] Setting key e4056968d7939d84b8bfc366d7512b63
I20260925 00:52:18.516927 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.508841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:56361 (local address 127.0.249.126:46377)
0925 00:52:18.508973 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.508984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.510194 (+ 1210us) server_negotiation.cc:407] Connection header received
0925 00:52:18.512036 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.512040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.512090 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.512177 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:18.513171 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.513746 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.514908 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.515287 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.515408 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.516537 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:18.516540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:18.516542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:18.516553 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:18.516572 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:18.516579 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:18.516638 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.516773 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.516822 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:52:18.517834 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.509223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46377 (local address 127.0.249.67:56361)
0925 00:52:18.509999 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:18.510058 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:18.511814 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:18.512320 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:18.512326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:18.512429 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:18.513025 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.513039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.513869 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.513872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.514735 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.514743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.515469 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.515476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.515747 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.516307 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:52:18.516308 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:18.516346 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:52:18.516405 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:18.516654 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:18.516660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:18.516664 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 00:52:18.516755 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 00:52:18.516955 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":151,"threads_started":1}
I20260925 00:52:18.518826 19035 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46377
I20260925 00:52:18.519006 19035 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:18.519443 19035 heartbeater.cc:511] Master 127.0.249.126:46377 requested a full tablet report, sending...
I20260925 00:52:18.520323 18584 ts_manager.cc:194] Registered new tserver with Master: 0a728ba61d02468bb75bf90cb7a7254c (127.0.249.67:36833)
I20260925 00:52:18.520922 18584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.67:56361
I20260925 00:52:18.524150 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:52:18.529328 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.526070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.0.249.126:46377)
0925 00:52:18.526211 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.526215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.526361 (+ 146us) server_negotiation.cc:407] Connection header received
0925 00:52:18.526445 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.526447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.526482 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.526543 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:18.526783 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.527258 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.528018 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.528215 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.528286 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.528962 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:18.528965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:18.528967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:18.528977 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:18.528993 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:18.529000 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:18.529055 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.529138 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.529180 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 00:52:18.531836 18584 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:52:18.534705 18584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:52:18.549759 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.542962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:35465 (local address 127.0.0.1:54368)
0925 00:52:18.543528 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:18.543580 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:18.543708 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:18.544877 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:18.544881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:18.544899 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:18.545017 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.545022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.546534 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.546538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.547966 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.547976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.549309 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.549312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.549412 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.549435 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:52:18.549507 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:52:18.549570 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":61,"thread_start_us":115,"threads_started":1}
I20260925 00:52:18.549825 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.544015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.0.249.67:36833)
0925 00:52:18.544386 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.544395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.544414 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:52:18.544473 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.544477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.544623 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.544753 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:18.545119 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.546177 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.548041 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.548824 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.548929 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.549106 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.549554 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.549664 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":171,"threads_started":1}
I20260925 00:52:18.550274 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.543665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.0.249.65:35465)
0925 00:52:18.544463 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.544471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.544489 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:52:18.544549 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.544552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.544698 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.544870 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:18.545216 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.546403 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.548116 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.548954 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.549059 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.549252 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.550020 (+ 768us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.550135 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":106,"threads_started":1}
I20260925 00:52:18.549926 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.542805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:34769 (local address 127.0.0.1:53810)
0925 00:52:18.543528 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:18.543580 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:18.543708 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:18.545611 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:18.545614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:18.545633 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:18.545711 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.545717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.547253 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.547257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.548353 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.548363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.549454 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.549458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.549568 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.549591 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:52:18.549651 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 00:52:18.549735 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"spinlock_wait_cycles":28544,"thread_start_us":64,"threads_started":1}
I20260925 00:52:18.550109 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.543193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:36833 (local address 127.0.0.1:57792)
0925 00:52:18.544020 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:18.544033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:18.544120 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:18.544786 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:18.544795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:18.544823 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:18.544980 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.544990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.546293 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.546297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.547911 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.547921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.548940 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.548944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.549098 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.549140 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 00:52:18.549244 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 00:52:18.549355 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":44,"thread_start_us":94,"threads_started":1}
I20260925 00:52:18.550614 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.542953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.0.249.66:34769)
0925 00:52:18.545158 (+ 2205us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.545187 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.545210 (+ 23us) server_negotiation.cc:407] Connection header received
0925 00:52:18.545280 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.545284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.545474 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.545619 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:18.545828 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.547138 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.548503 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.549287 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.550004 (+ 717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.550190 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.550230 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.550295 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2089,"thread_start_us":125,"threads_started":1}
I20260925 00:52:18.553496 18707 tablet_service.cc:1467] Processing CreateTablet for tablet b7e12a4a0d4f45218cc2fdf96ab4c379 (DEFAULT_TABLE table=test-table [id=543efcce594647c1a7a09fcacdb2c29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:18.553495 18838 tablet_service.cc:1467] Processing CreateTablet for tablet b7e12a4a0d4f45218cc2fdf96ab4c379 (DEFAULT_TABLE table=test-table [id=543efcce594647c1a7a09fcacdb2c29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:18.554582 18707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e12a4a0d4f45218cc2fdf96ab4c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:18.554582 18838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e12a4a0d4f45218cc2fdf96ab4c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:18.555860 18969 tablet_service.cc:1467] Processing CreateTablet for tablet b7e12a4a0d4f45218cc2fdf96ab4c379 (DEFAULT_TABLE table=test-table [id=543efcce594647c1a7a09fcacdb2c29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:18.556668 18969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e12a4a0d4f45218cc2fdf96ab4c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:18.561801 19052 tablet_bootstrap.cc:492] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: Bootstrap starting.
I20260925 00:52:18.562225 19053 tablet_bootstrap.cc:492] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386: Bootstrap starting.
I20260925 00:52:18.563014 19054 tablet_bootstrap.cc:492] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: Bootstrap starting.
I20260925 00:52:18.564851 19052 tablet_bootstrap.cc:654] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:18.564870 19054 tablet_bootstrap.cc:654] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:18.565599 19054 log.cc:824] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:18.565623 19052 log.cc:824] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:18.566941 19053 tablet_bootstrap.cc:654] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:18.567500 19054 tablet_bootstrap.cc:492] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: No bootstrap required, opened a new log
I20260925 00:52:18.567502 19052 tablet_bootstrap.cc:492] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: No bootstrap required, opened a new log
I20260925 00:52:18.567596 19053 log.cc:824] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:18.567685 19054 ts_tablet_manager.cc:1397] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:52:18.567685 19052 ts_tablet_manager.cc:1397] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260925 00:52:18.569320 19053 tablet_bootstrap.cc:492] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386: No bootstrap required, opened a new log
I20260925 00:52:18.569533 19053 ts_tablet_manager.cc:1397] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 00:52:18.571942 19054 raft_consensus.cc:348] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } }
I20260925 00:52:18.571942 19052 raft_consensus.cc:348] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } }
I20260925 00:52:18.572206 19054 raft_consensus.cc:374] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:18.572206 19052 raft_consensus.cc:374] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:18.572273 19052 raft_consensus.cc:729] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9296451d6e941c6b9de1e31e4a19487, State: Initialized, Role: FOLLOWER
I20260925 00:52:18.572273 19054 raft_consensus.cc:729] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a728ba61d02468bb75bf90cb7a7254c, State: Initialized, Role: FOLLOWER
I20260925 00:52:18.572773 19054 consensus_queue.cc:260] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } }
I20260925 00:52:18.572773 19052 consensus_queue.cc:260] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } }
I20260925 00:52:18.573140 19053 raft_consensus.cc:348] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } }
I20260925 00:52:18.573405 19053 raft_consensus.cc:374] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:18.573468 18904 heartbeater.cc:503] Master 127.0.249.126:46377 was elected leader, sending a full tablet report...
I20260925 00:52:18.573472 19053 raft_consensus.cc:729] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5641c9be81846aaa5c393fb4b818386, State: Initialized, Role: FOLLOWER
I20260925 00:52:18.573803 19054 ts_tablet_manager.cc:1428] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:52:18.573892 19035 heartbeater.cc:503] Master 127.0.249.126:46377 was elected leader, sending a full tablet report...
I20260925 00:52:18.573923 19053 consensus_queue.cc:260] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } }
I20260925 00:52:18.574522 18773 heartbeater.cc:503] Master 127.0.249.126:46377 was elected leader, sending a full tablet report...
I20260925 00:52:18.574800 19052 ts_tablet_manager.cc:1428] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260925 00:52:18.575004 19053 ts_tablet_manager.cc:1428] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 00:52:18.578801 18905 tablet.cc:2369] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 00:52:18.643765 18774 tablet.cc:2369] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:18.701560 19060 raft_consensus.cc:482] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:18.701764 19060 raft_consensus.cc:504] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } }
I20260925 00:52:18.702854 19060 leader_election.cc:288] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9296451d6e941c6b9de1e31e4a19487 (127.0.249.66:34769), 0a728ba61d02468bb75bf90cb7a7254c (127.0.249.67:36833)
I20260925 00:52:18.706944 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.703218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:48653 (local address 127.0.249.66:34769)
0925 00:52:18.703420 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.703425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.703490 (+ 65us) server_negotiation.cc:407] Connection header received
0925 00:52:18.703623 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.703626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.703697 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.703780 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:18.704083 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.704684 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.705990 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.706398 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.706524 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.706587 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.706742 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.706824 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 00:52:18.706955 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.703071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:34769 (local address 127.0.249.65:48653)
0925 00:52:18.703389 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:18.703404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:18.703515 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:18.703797 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:18.703800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:18.703818 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:18.703962 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.703969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.704794 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.704801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.705865 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.705874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.706523 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.706529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.706619 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.706641 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:52:18.706721 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 00:52:18.706774 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":27,"thread_start_us":109,"threads_started":1}
I20260925 00:52:18.707059 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.703221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:55945 (local address 127.0.249.67:36833)
0925 00:52:18.703390 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.703396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.703650 (+ 254us) server_negotiation.cc:407] Connection header received
0925 00:52:18.703797 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.703800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.703846 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.703920 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:18.704194 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.704763 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.706037 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.706441 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.706604 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.706648 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.706866 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.706933 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 00:52:18.707325 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.703087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:36833 (local address 127.0.249.65:55945)
0925 00:52:18.703529 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:18.703562 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:18.703663 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:18.703941 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:18.703943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:18.703957 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:18.704085 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.704091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.704876 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.704879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.705881 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:18.705890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.706612 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.706617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:18.706735 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.706759 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:52:18.706833 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 00:52:18.706895 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":34,"thread_start_us":74,"threads_started":1}
I20260925 00:52:18.707810 18858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e12a4a0d4f45218cc2fdf96ab4c379" candidate_uuid: "a5641c9be81846aaa5c393fb4b818386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9296451d6e941c6b9de1e31e4a19487" is_pre_election: true
I20260925 00:52:18.707887 18989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e12a4a0d4f45218cc2fdf96ab4c379" candidate_uuid: "a5641c9be81846aaa5c393fb4b818386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a728ba61d02468bb75bf90cb7a7254c" is_pre_election: true
I20260925 00:52:18.708129 18989 raft_consensus.cc:2454] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5641c9be81846aaa5c393fb4b818386 in term 0.
I20260925 00:52:18.708130 18858 raft_consensus.cc:2454] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5641c9be81846aaa5c393fb4b818386 in term 0.
I20260925 00:52:18.708626 18663 leader_election.cc:302] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a728ba61d02468bb75bf90cb7a7254c, a5641c9be81846aaa5c393fb4b818386, e9296451d6e941c6b9de1e31e4a19487; no voters:
I20260925 00:52:18.708909 19060 raft_consensus.cc:2781] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:52:18.709039 19060 raft_consensus.cc:482] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:52:18.709120 19060 raft_consensus.cc:3037] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:18.710219 19060 raft_consensus.cc:504] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } }
I20260925 00:52:18.710565 19060 leader_election.cc:288] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [CANDIDATE]: Term 1 election: Requested vote from peers e9296451d6e941c6b9de1e31e4a19487 (127.0.249.66:34769), 0a728ba61d02468bb75bf90cb7a7254c (127.0.249.67:36833)
I20260925 00:52:18.710889 18989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e12a4a0d4f45218cc2fdf96ab4c379" candidate_uuid: "a5641c9be81846aaa5c393fb4b818386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a728ba61d02468bb75bf90cb7a7254c"
I20260925 00:52:18.710889 18858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e12a4a0d4f45218cc2fdf96ab4c379" candidate_uuid: "a5641c9be81846aaa5c393fb4b818386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9296451d6e941c6b9de1e31e4a19487"
I20260925 00:52:18.711007 18989 raft_consensus.cc:3037] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:18.711007 18858 raft_consensus.cc:3037] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:18.712232 18858 raft_consensus.cc:2454] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5641c9be81846aaa5c393fb4b818386 in term 1.
I20260925 00:52:18.712232 18989 raft_consensus.cc:2454] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5641c9be81846aaa5c393fb4b818386 in term 1.
I20260925 00:52:18.712522 18661 leader_election.cc:302] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a728ba61d02468bb75bf90cb7a7254c, a5641c9be81846aaa5c393fb4b818386, e9296451d6e941c6b9de1e31e4a19487; no voters:
I20260925 00:52:18.712710 19060 raft_consensus.cc:2781] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:18.712877 19060 raft_consensus.cc:686] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [term 1 LEADER]: Becoming Leader. State: Replica: a5641c9be81846aaa5c393fb4b818386, State: Running, Role: LEADER
I20260925 00:52:18.713192 19060 consensus_queue.cc:237] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } }
I20260925 00:52:18.716564 18585 catalog_manager.cc:5495] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5641c9be81846aaa5c393fb4b818386 (127.0.249.65). New cstate: current_term: 1 leader_uuid: "a5641c9be81846aaa5c393fb4b818386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5641c9be81846aaa5c393fb4b818386" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 } health_report { overall_health: UNKNOWN } } }
I20260925 00:52:18.735654 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.732520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.0.249.65:35465)
0925 00:52:18.732658 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:18.732661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:18.732673 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:52:18.732743 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:18.732745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:18.732798 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:18.732883 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:18.733079 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.733638 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.734433 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:18.734640 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:18.734714 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:18.734956 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:18.735367 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:18.735454 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:18.735507 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 00:52:18.735550 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:52:18.741310 18858 raft_consensus.cc:1261] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [term 1 FOLLOWER]: Refusing update from remote peer a5641c9be81846aaa5c393fb4b818386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:18.741310 18989 raft_consensus.cc:1261] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [term 1 FOLLOWER]: Refusing update from remote peer a5641c9be81846aaa5c393fb4b818386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:18.742043 19063 consensus_queue.cc:1035] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a728ba61d02468bb75bf90cb7a7254c" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:18.742262 19060 consensus_queue.cc:1035] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9296451d6e941c6b9de1e31e4a19487" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:18.750383 19067 mvcc.cc:205] Tried to move back new op lower bound from 7333058718677647360 to 7333058718572036096. Current Snapshot: MvccSnapshot[applied={T|T < 7333058718677647360}]
I20260925 00:52:18.757874 18585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:18.758044 18585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:18.760406 18585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8169ab534964f1899a8a9a7bb04ee76: Sending DeleteTablet for 3 replicas of tablet b7e12a4a0d4f45218cc2fdf96ab4c379
I20260925 00:52:18.761165 18707 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e12a4a0d4f45218cc2fdf96ab4c379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:18 UTC) from {username='slave'} at 127.0.0.1:54368
I20260925 00:52:18.761430 18969 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e12a4a0d4f45218cc2fdf96ab4c379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:18 UTC) from {username='slave'} at 127.0.0.1:57792
I20260925 00:52:18.761677 18838 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e12a4a0d4f45218cc2fdf96ab4c379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:18 UTC) from {username='slave'} at 127.0.0.1:53810
I20260925 00:52:18.762140 19076 tablet_replica.cc:330] T b7e12a4a0d4f45218cc2fdf96ab4c379 P a5641c9be81846aaa5c393fb4b818386: stopping tablet replica
I20260925 00:52:18.762235 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 18645
I20260925 00:52:18.762874 19077 tablet_replica.cc:330] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: stopping tablet replica
I20260925 00:52:18.763177 19077 raft_consensus.cc:2229] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:52:18.763473 19077 raft_consensus.cc:2258] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:18.764617 19078 tablet_replica.cc:330] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: stopping tablet replica
I20260925 00:52:18.764842 19077 ts_tablet_manager.cc:1905] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:18.764946 19078 raft_consensus.cc:2229] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:52:18.765244 19078 raft_consensus.cc:2258] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:18.766716 19078 ts_tablet_manager.cc:1905] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:18.767239 19077 ts_tablet_manager.cc:1918] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:18.767349 19077 log.cc:1196] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/b7e12a4a0d4f45218cc2fdf96ab4c379
I20260925 00:52:18.767639 19077 ts_tablet_manager.cc:1939] T b7e12a4a0d4f45218cc2fdf96ab4c379 P 0a728ba61d02468bb75bf90cb7a7254c: Deleting consensus metadata
I20260925 00:52:18.769102 18570 catalog_manager.cc:4841] TS 0a728ba61d02468bb75bf90cb7a7254c (127.0.249.67:36833): tablet b7e12a4a0d4f45218cc2fdf96ab4c379 (table test-table [id=543efcce594647c1a7a09fcacdb2c29e]) successfully deleted
I20260925 00:52:18.770004 19078 ts_tablet_manager.cc:1918] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:18.770098 19078 log.cc:1196] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/b7e12a4a0d4f45218cc2fdf96ab4c379
I20260925 00:52:18.770423 19078 ts_tablet_manager.cc:1939] T b7e12a4a0d4f45218cc2fdf96ab4c379 P e9296451d6e941c6b9de1e31e4a19487: Deleting consensus metadata
W20260925 00:52:18.770891 18569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:35465 (error 108)
I20260925 00:52:18.771590 18572 catalog_manager.cc:4841] TS e9296451d6e941c6b9de1e31e4a19487 (127.0.249.66:34769): tablet b7e12a4a0d4f45218cc2fdf96ab4c379 (table test-table [id=543efcce594647c1a7a09fcacdb2c29e]) successfully deleted
I20260925 00:52:18.771692 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 18776
I20260925 00:52:18.772188 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:18.771840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:35465 (local address 127.0.0.1:54382)
0925 00:52:18.772024 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:18.772088 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260925 00:52:18.772363 18569 catalog_manager.cc:4584] TS a5641c9be81846aaa5c393fb4b818386 (127.0.249.65:35465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7e12a4a0d4f45218cc2fdf96ab4c379: Network error: Client connection negotiation failed: client connection to 127.0.249.65:35465: connect: Connection refused (error 111)
I20260925 00:52:18.777900 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 18907
I20260925 00:52:18.783176 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 18554
2026-09-25T00:52:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1083 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:18 dist-test-slave-05m0 krb5kdc[19082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:18 dist-test-slave-05m0 krb5kdc[19082](info): set up 2 sockets
Sep 25 00:52:18 dist-test-slave-05m0 krb5kdc[19082](info): commencing operation
krb5kdc: starting...
W20260925 00:52:19.825284 997 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:19 dist-test-slave-05m0 krb5kdc[19082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:19.944795 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46591
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:20.052057 19095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:20.052392 19095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:20.052479 19095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:20.056452 19095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:20.056538 19095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:20.056577 19095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:20.056614 19095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:20.056646 19095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:20.061805 19095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:20.062860 19095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:20.063872 19095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:20.072502 19101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:20.072482 19100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:20.072479 19103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:20.072518 19095 server_base.cc:1034] running on GCE node
I20260925 00:52:20.073475 19095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:20.074644 19095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:20.075830 19095 hybrid_clock.cc:648] HybridClock initialized: now 1790297540075797 us; error 47 us; skew 500 ppm
Sep 25 00:52:20 dist-test-slave-05m0 krb5kdc[19082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:20.083842 19095 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:20.084833 19095 webserver.cc:460] Webserver started at http://127.0.249.126:40825/ using document root <none> and password file <none>
I20260925 00:52:20.085481 19095 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:20.085574 19095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:20.085810 19095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:20.087815 19095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "e9287375e6ab41c4938b43de0339e45f"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "962c00b1116bf539b1e47dfafc75256d"
server_key_iv: "c7582a66a940f1d9ac6f78191727c9fc"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.088393 19095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "e9287375e6ab41c4938b43de0339e45f"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "962c00b1116bf539b1e47dfafc75256d"
server_key_iv: "c7582a66a940f1d9ac6f78191727c9fc"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.091984 19095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 00:52:20.094408 19110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.095646 19095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:20.095783 19095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "e9287375e6ab41c4938b43de0339e45f"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "962c00b1116bf539b1e47dfafc75256d"
server_key_iv: "c7582a66a940f1d9ac6f78191727c9fc"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.095896 19095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:20.105309 19095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:20.108008 19095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:20.108211 19095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:20.115170 19095 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:46591
I20260925 00:52:20.115186 19162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:46591 every 8 connection(s)
I20260925 00:52:20.116081 19095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:20.118762 19163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:20.120508 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 19095
I20260925 00:52:20.120609 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:20.120954 997 external_mini_cluster.cc:1428] Setting key bc062a9b3b41df139bce57d0d65f0f47
I20260925 00:52:20.125025 19163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f: Bootstrap starting.
I20260925 00:52:20.127228 19163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:20.127923 19163 log.cc:824] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:52:20 dist-test-slave-05m0 krb5kdc[19082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:20.130095 19163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f: No bootstrap required, opened a new log
I20260925 00:52:20.134893 19163 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9287375e6ab41c4938b43de0339e45f" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46591 } }
I20260925 00:52:20.135169 19163 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:20.135217 19163 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9287375e6ab41c4938b43de0339e45f, State: Initialized, Role: FOLLOWER
I20260925 00:52:20.135236 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.122408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.0.249.126:46591)
0925 00:52:20.122837 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.122851 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.122885 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:52:20.123518 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.123534 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.123847 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.124183 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:20.124972 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.126052 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.126931 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.127464 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.127628 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.130325 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:20.130331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:20.130346 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:20.130376 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:20.132289 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.132702 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.132709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.132714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.132845 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.133095 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.133098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.133100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.133599 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:20.133792 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.134001 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.134292 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":135,"threads_started":1}
I20260925 00:52:20.135676 19163 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9287375e6ab41c4938b43de0339e45f" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46591 } }
I20260925 00:52:20.135828 19163 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:20.135890 19163 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:20.135993 19163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:20.136847 19163 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9287375e6ab41c4938b43de0339e45f" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46591 } }
I20260925 00:52:20.137228 19163 leader_election.cc:302] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9287375e6ab41c4938b43de0339e45f; no voters:
I20260925 00:52:20.137593 19163 leader_election.cc:288] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:20.137697 19168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:20.137897 19168 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [term 1 LEADER]: Becoming Leader. State: Replica: e9287375e6ab41c4938b43de0339e45f, State: Running, Role: LEADER
I20260925 00:52:20.138204 19168 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9287375e6ab41c4938b43de0339e45f" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46591 } }
I20260925 00:52:20.138773 19163 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:20.139703 19170 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9287375e6ab41c4938b43de0339e45f. Latest consensus state: current_term: 1 leader_uuid: "e9287375e6ab41c4938b43de0339e45f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9287375e6ab41c4938b43de0339e45f" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46591 } } }
I20260925 00:52:20.139840 19170 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:20.140190 19177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:20.140803 19169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9287375e6ab41c4938b43de0339e45f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9287375e6ab41c4938b43de0339e45f" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46591 } } }
I20260925 00:52:20.140911 19169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:20.143219 19177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:20.148356 19177 catalog_manager.cc:1347] Generated new cluster ID: a860031a3d1847fab41fe7e09a8eb03c
I20260925 00:52:20.148427 19177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:20.180763 19177 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:20.181833 19177 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:20.186774 19177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f: Generated new TSK 0
I20260925 00:52:20.187486 19177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:52:20.228368 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46591
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:20.362442 19189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:20.362700 19189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:20.362746 19189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:20.367177 19189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:20.367233 19189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:20.367293 19189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:52:20.372874 19189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:20.373947 19189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:20.375020 19189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:20.384543 19197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:20.384565 19195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:20.384543 19194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:20.384995 19189 server_base.cc:1034] running on GCE node
I20260925 00:52:20.385318 19189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:20.385953 19189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:20.387108 19189 hybrid_clock.cc:648] HybridClock initialized: now 1790297540387074 us; error 47 us; skew 500 ppm
Sep 25 00:52:20 dist-test-slave-05m0 krb5kdc[19082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:20.389843 19189 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:52:20.390627 19189 webserver.cc:460] Webserver started at http://127.0.249.65:34215/ using document root <none> and password file <none>
I20260925 00:52:20.391119 19189 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:20.391183 19189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:20.391338 19189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:20.393244 19189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "6cbea498f73f4af59fea598b19ef1e7e"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "fdc3e4e91878d040495211192cc1d1b8"
server_key_iv: "5a53d6f72c27592cae72872f6de3b9c3"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.393808 19189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "6cbea498f73f4af59fea598b19ef1e7e"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "fdc3e4e91878d040495211192cc1d1b8"
server_key_iv: "5a53d6f72c27592cae72872f6de3b9c3"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.397714 19189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 00:52:20.400058 19204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.401242 19189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:20.401372 19189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "6cbea498f73f4af59fea598b19ef1e7e"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "fdc3e4e91878d040495211192cc1d1b8"
server_key_iv: "5a53d6f72c27592cae72872f6de3b9c3"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.401539 19189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:20.421795 19189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:20.425158 19189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:20.425366 19189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:20.425994 19189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:20.426832 19189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:20.426903 19189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.426972 19189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:20.427016 19189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.435907 19189 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:42281
I20260925 00:52:20.435926 19317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:42281 every 8 connection(s)
I20260925 00:52:20.436872 19189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:52:20.445374 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 19189
I20260925 00:52:20.445544 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:52:20.445822 997 external_mini_cluster.cc:1428] Setting key d7e9cec33252fa6a63783b3306ebfb92
Sep 25 00:52:20 dist-test-slave-05m0 krb5kdc[19082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:20.451697 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.438732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:55055 (local address 127.0.249.126:46591)
0925 00:52:20.438883 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.438887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.439764 (+ 877us) server_negotiation.cc:407] Connection header received
0925 00:52:20.441687 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.441689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.441736 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.441811 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:20.442956 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.443473 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.444679 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.444903 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.444985 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.447809 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:20.447815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:20.447821 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:20.447854 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:20.449498 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.450114 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.450116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.450118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.450202 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.450602 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.450604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.450605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.450768 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:20.450846 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.451454 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.451556 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:52:20.452930 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.439154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46591 (local address 127.0.249.65:55055)
0925 00:52:20.439640 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:20.439683 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:20.441463 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:20.441942 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:20.441949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:20.442318 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:20.442819 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.442833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.443605 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.443608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.444547 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.444557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.445060 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.445063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.445211 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.445944 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:20.445969 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:20.447638 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:20.449607 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:20.449614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:20.449631 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:20.449998 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:20.450293 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:20.450297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:20.450299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:20.450487 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:20.450873 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:20.450883 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:20.451210 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 00:52:20.451423 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 00:52:20.451794 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":111,"threads_started":1}
I20260925 00:52:20.454222 19318 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46591
I20260925 00:52:20.454455 19318 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:20.454933 19318 heartbeater.cc:511] Master 127.0.249.126:46591 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:52:20.456423 19127 ts_manager.cc:194] Registered new tserver with Master: 6cbea498f73f4af59fea598b19ef1e7e (127.0.249.65:42281)
I20260925 00:52:20.457763 19127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:55055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:52:20.473237 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46591
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:20.582510 19323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:20.582757 19323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:20.582800 19323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:20.586761 19323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:20.586817 19323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:20.586874 19323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:52:20.592006 19323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:20.592971 19323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:20.594000 19323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:20.603619 19328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:20.603564 19329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:20.603560 19331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:20.603785 19323 server_base.cc:1034] running on GCE node
I20260925 00:52:20.604393 19323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:20.605031 19323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:20.606236 19323 hybrid_clock.cc:648] HybridClock initialized: now 1790297540606192 us; error 55 us; skew 500 ppm
Sep 25 00:52:20 dist-test-slave-05m0 krb5kdc[19082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:20.608906 19323 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:52:20.609771 19323 webserver.cc:460] Webserver started at http://127.0.249.66:35007/ using document root <none> and password file <none>
I20260925 00:52:20.610361 19323 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:20.610451 19323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:20.610678 19323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:20.612651 19323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "ab935324bc594b3f9138fbaa9d43547a"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "08650388129fa1e155a2da350401a4e1"
server_key_iv: "4ee78740617e08afc466f284f3a106e7"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.613205 19323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "ab935324bc594b3f9138fbaa9d43547a"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "08650388129fa1e155a2da350401a4e1"
server_key_iv: "4ee78740617e08afc466f284f3a106e7"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.616778 19323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:52:20.619167 19338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.620322 19323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:52:20.620452 19323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "ab935324bc594b3f9138fbaa9d43547a"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "08650388129fa1e155a2da350401a4e1"
server_key_iv: "4ee78740617e08afc466f284f3a106e7"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.620558 19323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:20.632720 19323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:20.635483 19323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:20.635676 19323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:20.636214 19323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:20.636973 19323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:20.637042 19323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.637118 19323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:20.637161 19323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.646178 19323 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:42387
I20260925 00:52:20.646204 19451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:42387 every 8 connection(s)
I20260925 00:52:20.646993 19323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:52:20.649122 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 19323
I20260925 00:52:20.649240 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:52:20.649607 997 external_mini_cluster.cc:1428] Setting key 224f29a238b58bcb7f88f01f2e2b8ecb
Sep 25 00:52:20 dist-test-slave-05m0 krb5kdc[19082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:52:20.661240 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.648730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:49955 (local address 127.0.249.126:46591)
0925 00:52:20.648857 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.648860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.650235 (+ 1375us) server_negotiation.cc:407] Connection header received
0925 00:52:20.651652 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.651658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.651720 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.651813 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:20.652955 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.653825 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.654910 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.655133 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.655245 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.657674 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:20.657678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:20.657681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:20.657713 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:20.659352 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.659818 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.659820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.659821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.659894 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.660252 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.660254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.660255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.660424 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:20.660499 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.660951 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.661085 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
I20260925 00:52:20.662144 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.649208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46591 (local address 127.0.249.66:49955)
0925 00:52:20.650028 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:20.650122 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:20.651443 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:20.651962 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:20.651969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:20.652309 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:20.652812 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.652824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.653931 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.653934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.654784 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.654793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.655279 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.655283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.655449 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.656032 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:20.656048 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:20.657508 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:20.659466 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:20.659471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:20.659481 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:20.659733 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:20.659970 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:20.659972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:20.659973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:20.660167 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:20.660511 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:20.660516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:20.660747 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 00:52:20.660901 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 00:52:20.661176 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":155,"threads_started":1}
I20260925 00:52:20.663344 19452 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46591
I20260925 00:52:20.663592 19452 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:20.664064 19452 heartbeater.cc:511] Master 127.0.249.126:46591 requested a full tablet report, sending...
I20260925 00:52:20.665096 19127 ts_manager.cc:194] Registered new tserver with Master: ab935324bc594b3f9138fbaa9d43547a (127.0.249.66:42387)
I20260925 00:52:20.665989 19127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:49955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:52:20.676638 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46591
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:20.785916 19457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:20.786150 19457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:20.786192 19457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:20.790637 19457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:20.790692 19457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:20.790750 19457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:52:20.795837 19457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:46591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:20.796727 19457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:20.797703 19457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:20.807263 19465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:20.807277 19462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:20.807277 19463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:20.807312 19457 server_base.cc:1034] running on GCE node
I20260925 00:52:20.808005 19457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:20.808736 19457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:20.809898 19457 hybrid_clock.cc:648] HybridClock initialized: now 1790297540809874 us; error 37 us; skew 500 ppm
Sep 25 00:52:20 dist-test-slave-05m0 krb5kdc[19082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:20.812533 19457 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:52:20.813341 19457 webserver.cc:460] Webserver started at http://127.0.249.67:36299/ using document root <none> and password file <none>
I20260925 00:52:20.813930 19457 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:20.814019 19457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:20.814247 19457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:20.816196 19457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "e5aa1537434347ea8d0af28d9c50fdbe"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "561194882dca424aba3d6ba3096430c2"
server_key_iv: "cfed9386e63b224b9f26d9bdf3f07730"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.816761 19457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "e5aa1537434347ea8d0af28d9c50fdbe"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "561194882dca424aba3d6ba3096430c2"
server_key_iv: "cfed9386e63b224b9f26d9bdf3f07730"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.820283 19457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:52:20.822664 19472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.823868 19457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:52:20.823997 19457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "e5aa1537434347ea8d0af28d9c50fdbe"
format_stamp: "Formatted at 2026-09-25 00:52:20 on dist-test-slave-05m0"
server_key: "561194882dca424aba3d6ba3096430c2"
server_key_iv: "cfed9386e63b224b9f26d9bdf3f07730"
server_key_version: "encryptionkey@0"
I20260925 00:52:20.824108 19457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:20.841959 19457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:20.845067 19457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:20.845257 19457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:20.845870 19457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:20.846669 19457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:20.846738 19457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.846805 19457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:20.846851 19457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:20.855319 19457 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:36169
I20260925 00:52:20.855372 19585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:36169 every 8 connection(s)
I20260925 00:52:20.856484 19457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:52:20.862525 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 19457
I20260925 00:52:20.862670 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:52:20.862995 997 external_mini_cluster.cc:1428] Setting key 7c3bbea207e0686090174189234e1ae8
Sep 25 00:52:20 dist-test-slave-05m0 krb5kdc[19082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:20.869705 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.858250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:32917 (local address 127.0.249.126:46591)
0925 00:52:20.858397 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.858400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.859091 (+ 691us) server_negotiation.cc:407] Connection header received
0925 00:52:20.860323 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.860325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.860372 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.860449 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:20.861632 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.862133 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.863744 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.864058 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.864205 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.866655 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:20.866658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:20.866660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:20.866684 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:20.867893 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.868331 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.868333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.868334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.868404 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.868680 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.868682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.868683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.868841 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:20.868914 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.869387 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.869530 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 00:52:20.870545 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.858512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:46591 (local address 127.0.249.67:32917)
0925 00:52:20.858927 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:20.858979 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:20.860138 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:20.860650 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:20.860659 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:20.861025 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:20.861508 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.861522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.862241 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.862243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.863581 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.863591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.864260 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.864266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.864452 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.865006 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:20.865047 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:20.866501 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:20.867991 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:20.867995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:20.868005 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:20.868249 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:20.868480 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:20.868482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:20.868484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:20.868605 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:20.868924 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:20.868929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:20.869130 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 00:52:20.869337 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 00:52:20.869674 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":117,"threads_started":1}
I20260925 00:52:20.871598 19586 heartbeater.cc:347] Connected to a master server at 127.0.249.126:46591
I20260925 00:52:20.871804 19586 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:20.872216 19586 heartbeater.cc:511] Master 127.0.249.126:46591 requested a full tablet report, sending...
I20260925 00:52:20.873135 19127 ts_manager.cc:194] Registered new tserver with Master: e5aa1537434347ea8d0af28d9c50fdbe (127.0.249.67:36169)
I20260925 00:52:20.873708 19127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:32917
I20260925 00:52:20.876573 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:52:20.885342 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.878473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.0.249.126:46591)
0925 00:52:20.878591 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.878596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.878792 (+ 196us) server_negotiation.cc:407] Connection header received
0925 00:52:20.878961 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.878963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.879003 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.879066 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:20.879699 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.880185 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.880949 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.881152 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.881222 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.882721 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:20.882724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:20.882726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:20.882746 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:20.883895 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.884344 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.884346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.884348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.884414 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:20.884666 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:20.884668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:20.884669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:20.884840 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:20.884943 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.885165 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.885235 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:52:20.888048 19127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:52:20.891003 19127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:52:20.904572 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.898597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:36169 (local address 127.0.0.1:36528)
0925 00:52:20.899276 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:20.899325 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:20.899653 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:20.900379 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:20.900385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:20.900408 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:20.900593 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.900600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.901774 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.901780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.903073 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.903084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.903999 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.904003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.904105 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.904141 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 00:52:20.904301 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 00:52:20.904416 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260925 00:52:20.904618 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.898939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:42387 (local address 127.0.0.1:58266)
0925 00:52:20.899430 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:20.899444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:20.899522 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:20.900399 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:20.900402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:20.900420 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:20.900612 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.900621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.902144 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.902147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.903278 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.903286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.904294 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.904298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.904378 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.904398 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 00:52:20.904449 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 00:52:20.904510 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":75,"spinlock_wait_cycles":5504,"thread_start_us":76,"threads_started":1}
I20260925 00:52:20.904745 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.899383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.0.249.66:42387)
0925 00:52:20.899918 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.899929 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.899950 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:52:20.900024 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.900028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.900243 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.900736 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:20.900896 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.901995 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.903389 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.904166 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.904291 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.904424 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.904488 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.904569 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":103,"threads_started":1}
I20260925 00:52:20.905202 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.898844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:42281 (local address 127.0.0.1:39360)
0925 00:52:20.899409 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:20.899422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:20.899522 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:20.900545 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:20.900549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:20.900566 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:20.900685 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.900693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.902179 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.902182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.903178 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.903191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.904751 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.904754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.904916 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.904944 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 00:52:20.905014 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 00:52:20.905079 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":4,"spinlock_wait_cycles":5248,"thread_start_us":93,"threads_started":1}
I20260925 00:52:20.905308 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.898770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.0.249.67:36169)
0925 00:52:20.899798 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.899810 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.899834 (+ 24us) server_negotiation.cc:407] Connection header received
0925 00:52:20.899929 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.899936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.900146 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.900748 (+ 602us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:20.900882 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.901646 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.903209 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.903878 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.903999 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.904146 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.905024 (+ 878us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.905135 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":126,"threads_started":1}
I20260925 00:52:20.905714 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.899736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.0.249.65:42281)
0925 00:52:20.900120 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.900132 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.900151 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:52:20.900229 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.900233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.900401 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.900553 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:20.900775 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.901995 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.903287 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.904166 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.904291 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.904424 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.905460 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.905557 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":120,"threads_started":1}
I20260925 00:52:20.907740 19520 tablet_service.cc:1467] Processing CreateTablet for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880 (DEFAULT_TABLE table=test-table [id=0b37cf8c1b1d49fbb53a6c5e9e738a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:20.907744 19386 tablet_service.cc:1467] Processing CreateTablet for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880 (DEFAULT_TABLE table=test-table [id=0b37cf8c1b1d49fbb53a6c5e9e738a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:20.908655 19520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:20.908370 19252 tablet_service.cc:1467] Processing CreateTablet for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880 (DEFAULT_TABLE table=test-table [id=0b37cf8c1b1d49fbb53a6c5e9e738a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:20.908730 19386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:20.909204 19252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:20.914386 19603 tablet_bootstrap.cc:492] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: Bootstrap starting.
I20260925 00:52:20.916193 19605 tablet_bootstrap.cc:492] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: Bootstrap starting.
I20260925 00:52:20.917227 19603 tablet_bootstrap.cc:654] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:20.917694 19604 tablet_bootstrap.cc:492] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e: Bootstrap starting.
I20260925 00:52:20.918507 19603 log.cc:824] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:20.918864 19605 tablet_bootstrap.cc:654] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:20.919462 19604 tablet_bootstrap.cc:654] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:20.919655 19605 log.cc:824] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:20.920593 19604 log.cc:824] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:20.920646 19603 tablet_bootstrap.cc:492] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: No bootstrap required, opened a new log
I20260925 00:52:20.920989 19603 ts_tablet_manager.cc:1397] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260925 00:52:20.921797 19605 tablet_bootstrap.cc:492] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: No bootstrap required, opened a new log
I20260925 00:52:20.922029 19605 ts_tablet_manager.cc:1397] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:52:20.922762 19604 tablet_bootstrap.cc:492] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e: No bootstrap required, opened a new log
I20260925 00:52:20.922945 19604 ts_tablet_manager.cc:1397] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 00:52:20.925050 19603 raft_consensus.cc:348] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } }
I20260925 00:52:20.925330 19603 raft_consensus.cc:374] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:20.925438 19603 raft_consensus.cc:729] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5aa1537434347ea8d0af28d9c50fdbe, State: Initialized, Role: FOLLOWER
I20260925 00:52:20.925773 19605 raft_consensus.cc:348] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } }
I20260925 00:52:20.925901 19603 consensus_queue.cc:260] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } }
I20260925 00:52:20.926018 19605 raft_consensus.cc:374] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:20.926105 19605 raft_consensus.cc:729] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab935324bc594b3f9138fbaa9d43547a, State: Initialized, Role: FOLLOWER
I20260925 00:52:20.926108 19604 raft_consensus.cc:348] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } }
I20260925 00:52:20.926391 19604 raft_consensus.cc:374] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:20.926455 19604 raft_consensus.cc:729] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cbea498f73f4af59fea598b19ef1e7e, State: Initialized, Role: FOLLOWER
I20260925 00:52:20.926517 19586 heartbeater.cc:503] Master 127.0.249.126:46591 was elected leader, sending a full tablet report...
I20260925 00:52:20.926880 19603 ts_tablet_manager.cc:1428] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260925 00:52:20.926904 19604 consensus_queue.cc:260] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } }
I20260925 00:52:20.927191 19605 consensus_queue.cc:260] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } }
I20260925 00:52:20.927649 19318 heartbeater.cc:503] Master 127.0.249.126:46591 was elected leader, sending a full tablet report...
I20260925 00:52:20.927894 19604 ts_tablet_manager.cc:1428] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 00:52:20.928997 19452 heartbeater.cc:503] Master 127.0.249.126:46591 was elected leader, sending a full tablet report...
I20260925 00:52:20.929692 19605 ts_tablet_manager.cc:1428] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260925 00:52:20.937939 19319 tablet.cc:2369] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:20.964457 19609 raft_consensus.cc:482] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:20.964627 19609 raft_consensus.cc:504] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } }
I20260925 00:52:20.965687 19609 leader_election.cc:288] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cbea498f73f4af59fea598b19ef1e7e (127.0.249.65:42281), ab935324bc594b3f9138fbaa9d43547a (127.0.249.66:42387)
I20260925 00:52:20.969385 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.965879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:42281 (local address 127.0.249.67:34603)
0925 00:52:20.966052 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:20.966067 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:20.966154 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:20.966479 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:20.966482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:20.966502 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:20.966640 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.966647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.967418 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.967421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.968337 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.968346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.968956 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.968963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.969064 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.969089 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:52:20.969165 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:52:20.969244 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":60}
I20260925 00:52:20.969430 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.965996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:34603 (local address 127.0.249.65:42281)
0925 00:52:20.966131 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.966135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.966150 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:52:20.966295 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.966301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.966364 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.966442 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:20.966764 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.967313 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.968465 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.968844 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.968931 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.969000 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.969209 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.969286 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:52:20.969448 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.965909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:42387 (local address 127.0.249.67:46323)
0925 00:52:20.966276 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:20.966290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:20.966388 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:20.966641 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:20.966643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:20.966663 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:20.966793 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.966800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.967515 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.967517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.968348 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:20.968356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.969032 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.969035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:20.969119 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.969156 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 00:52:20.969236 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 00:52:20.969296 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":19,"thread_start_us":119,"threads_started":1}
I20260925 00:52:20.969480 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.966044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:46323 (local address 127.0.249.66:42387)
0925 00:52:20.966267 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.966272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.966370 (+ 98us) server_negotiation.cc:407] Connection header received
0925 00:52:20.966493 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.966496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.966543 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.966625 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:20.966909 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.967412 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.968497 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:20.968907 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:20.969020 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:20.969069 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:20.969268 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 00:52:20.969339 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260925 00:52:20.970074 19406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c36d79cd0fc42b3b5fa67abeb7a6880" candidate_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab935324bc594b3f9138fbaa9d43547a" is_pre_election: true
I20260925 00:52:20.970098 19272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c36d79cd0fc42b3b5fa67abeb7a6880" candidate_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbea498f73f4af59fea598b19ef1e7e" is_pre_election: true
I20260925 00:52:20.970350 19406 raft_consensus.cc:2454] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5aa1537434347ea8d0af28d9c50fdbe in term 0.
I20260925 00:52:20.970362 19272 raft_consensus.cc:2454] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5aa1537434347ea8d0af28d9c50fdbe in term 0.
I20260925 00:52:20.970793 19473 leader_election.cc:302] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab935324bc594b3f9138fbaa9d43547a, e5aa1537434347ea8d0af28d9c50fdbe; no voters:
I20260925 00:52:20.971041 19609 raft_consensus.cc:2781] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:52:20.971122 19609 raft_consensus.cc:482] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:52:20.971163 19609 raft_consensus.cc:3037] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:20.972122 19609 raft_consensus.cc:504] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } }
I20260925 00:52:20.972443 19609 leader_election.cc:288] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [CANDIDATE]: Term 1 election: Requested vote from peers 6cbea498f73f4af59fea598b19ef1e7e (127.0.249.65:42281), ab935324bc594b3f9138fbaa9d43547a (127.0.249.66:42387)
I20260925 00:52:20.972766 19272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c36d79cd0fc42b3b5fa67abeb7a6880" candidate_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbea498f73f4af59fea598b19ef1e7e"
I20260925 00:52:20.972815 19406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c36d79cd0fc42b3b5fa67abeb7a6880" candidate_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab935324bc594b3f9138fbaa9d43547a"
I20260925 00:52:20.972874 19272 raft_consensus.cc:3037] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:20.972967 19406 raft_consensus.cc:3037] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:20.973984 19272 raft_consensus.cc:2454] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5aa1537434347ea8d0af28d9c50fdbe in term 1.
I20260925 00:52:20.974129 19406 raft_consensus.cc:2454] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5aa1537434347ea8d0af28d9c50fdbe in term 1.
I20260925 00:52:20.974305 19474 leader_election.cc:302] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cbea498f73f4af59fea598b19ef1e7e, e5aa1537434347ea8d0af28d9c50fdbe; no voters:
I20260925 00:52:20.974474 19609 raft_consensus.cc:2781] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:20.974694 19609 raft_consensus.cc:686] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 1 LEADER]: Becoming Leader. State: Replica: e5aa1537434347ea8d0af28d9c50fdbe, State: Running, Role: LEADER
I20260925 00:52:20.975121 19609 consensus_queue.cc:237] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } }
I20260925 00:52:20.978264 19127 catalog_manager.cc:5495] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe reported cstate change: term changed from 0 to 1, leader changed from <none> to e5aa1537434347ea8d0af28d9c50fdbe (127.0.249.67). New cstate: current_term: 1 leader_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5aa1537434347ea8d0af28d9c50fdbe" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 } health_report { overall_health: UNKNOWN } } }
I20260925 00:52:21.002758 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:20.999307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.0.249.67:36169)
0925 00:52:20.999439 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:20.999442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:20.999455 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:52:20.999517 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:20.999519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:20.999568 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:20.999659 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:20.999936 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.000503 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.001473 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.001680 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.001768 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.002021 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:21.002406 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:21.002515 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:21.002591 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 00:52:21.002625 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:52:21.008239 19272 raft_consensus.cc:1261] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e [term 1 FOLLOWER]: Refusing update from remote peer e5aa1537434347ea8d0af28d9c50fdbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:21.008239 19406 raft_consensus.cc:1261] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [term 1 FOLLOWER]: Refusing update from remote peer e5aa1537434347ea8d0af28d9c50fdbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:21.008935 19609 consensus_queue.cc:1035] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cbea498f73f4af59fea598b19ef1e7e" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 42281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:21.009132 19613 consensus_queue.cc:1035] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab935324bc594b3f9138fbaa9d43547a" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:21.016387 19617 mvcc.cc:205] Tried to move back new op lower bound from 7333058727963095040 to 7333058727837204480. Current Snapshot: MvccSnapshot[applied={T|T < 7333058727963095040}]
I20260925 00:52:21.017997 19618 mvcc.cc:205] Tried to move back new op lower bound from 7333058727963095040 to 7333058727837204480. Current Snapshot: MvccSnapshot[applied={T|T < 7333058727963095040}]
I20260925 00:52:21.023816 19126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:21.023952 19126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:21.026232 19126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9287375e6ab41c4938b43de0339e45f: Sending DeleteTablet for 3 replicas of tablet 5c36d79cd0fc42b3b5fa67abeb7a6880
I20260925 00:52:21.027012 19386 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:21 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:58266
I20260925 00:52:21.027017 19252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:21 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:39360
I20260925 00:52:21.027470 19626 tablet_replica.cc:330] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: stopping tablet replica
I20260925 00:52:21.027634 19520 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:21 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:36528
I20260925 00:52:21.027878 19626 raft_consensus.cc:2229] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:52:21.028242 19626 raft_consensus.cc:2258] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:21.028311 19625 tablet_replica.cc:330] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e: stopping tablet replica
I20260925 00:52:21.028434 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 19189
I20260925 00:52:21.028657 19625 raft_consensus.cc:2229] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P 6cbea498f73f4af59fea598b19ef1e7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:52:21.031049 19626 ts_tablet_manager.cc:1905] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:21.033154 19626 ts_tablet_manager.cc:1918] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:21.033262 19626 log.cc:1196] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/5c36d79cd0fc42b3b5fa67abeb7a6880
I20260925 00:52:21.033533 19627 tablet_replica.cc:330] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: stopping tablet replica
I20260925 00:52:21.033630 19626 ts_tablet_manager.cc:1939] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P ab935324bc594b3f9138fbaa9d43547a: Deleting consensus metadata
I20260925 00:52:21.034063 19627 raft_consensus.cc:2229] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:52:21.034451 19627 raft_consensus.cc:2258] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:21.034788 19111 catalog_manager.cc:4841] TS ab935324bc594b3f9138fbaa9d43547a (127.0.249.66:42387): tablet 5c36d79cd0fc42b3b5fa67abeb7a6880 (table test-table [id=0b37cf8c1b1d49fbb53a6c5e9e738a45]) successfully deleted
I20260925 00:52:21.035825 19627 ts_tablet_manager.cc:1905] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 00:52:21.036960 19112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:42281 (error 108)
I20260925 00:52:21.037997 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.037692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:42281 (local address 127.0.0.1:39372)
0925 00:52:21.037851 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:21.037917 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260925 00:52:21.038152 19112 catalog_manager.cc:4584] TS 6cbea498f73f4af59fea598b19ef1e7e (127.0.249.65:42281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c36d79cd0fc42b3b5fa67abeb7a6880: Network error: Client connection negotiation failed: client connection to 127.0.249.65:42281: connect: Connection refused (error 111)
I20260925 00:52:21.038309 19627 ts_tablet_manager.cc:1918] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:21.038425 19627 log.cc:1196] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/5c36d79cd0fc42b3b5fa67abeb7a6880
I20260925 00:52:21.038523 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 19323
I20260925 00:52:21.039868 19627 ts_tablet_manager.cc:1939] T 5c36d79cd0fc42b3b5fa67abeb7a6880 P e5aa1537434347ea8d0af28d9c50fdbe: Deleting consensus metadata
I20260925 00:52:21.041026 19113 catalog_manager.cc:4841] TS e5aa1537434347ea8d0af28d9c50fdbe (127.0.249.67:36169): tablet 5c36d79cd0fc42b3b5fa67abeb7a6880 (table test-table [id=0b37cf8c1b1d49fbb53a6c5e9e738a45]) successfully deleted
I20260925 00:52:21.045253 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 19457
I20260925 00:52:21.049875 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 19095
2026-09-25T00:52:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2267 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T00:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:21Z Disabled control of system clock
I20260925 00:52:21.079511 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:43035
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:52:21.187177 19632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:21.187426 19632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:21.187461 19632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:21.191339 19632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:21.191386 19632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:21.191401 19632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:21.191417 19632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:21.191432 19632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:21.195940 19632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:43035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:43035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:21.196894 19632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:21.198194 19632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:21.206471 19638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:21.206466 19640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:21.206467 19637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:21.206573 19632 server_base.cc:1034] running on GCE node
I20260925 00:52:21.207229 19632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:21.208250 19632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:21.209491 19632 hybrid_clock.cc:648] HybridClock initialized: now 1790297541209451 us; error 28 us; skew 500 ppm
I20260925 00:52:21.211087 19632 webserver.cc:460] Webserver started at http://127.0.249.126:45341/ using document root <none> and password file <none>
I20260925 00:52:21.211588 19632 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:21.211644 19632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:21.211798 19632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:21.213829 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "03bf96dba5704d93a40c66d7f5a38199"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "7caf93d69a7420ddedc5ce337b63683f"
server_key_iv: "22dd2d664f0c683512a4f18ddd870abb"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.214329 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "03bf96dba5704d93a40c66d7f5a38199"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "7caf93d69a7420ddedc5ce337b63683f"
server_key_iv: "22dd2d664f0c683512a4f18ddd870abb"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.217787 19632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:52:21.219986 19646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.221247 19632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:21.221349 19632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "03bf96dba5704d93a40c66d7f5a38199"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "7caf93d69a7420ddedc5ce337b63683f"
server_key_iv: "22dd2d664f0c683512a4f18ddd870abb"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.221522 19632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:21.248349 19632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:21.249022 19632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:21.249182 19632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:21.256430 19632 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:43035
I20260925 00:52:21.256453 19698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:43035 every 8 connection(s)
I20260925 00:52:21.257372 19632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:21.260428 19699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:21.264688 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 19632
I20260925 00:52:21.264813 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:21.265118 997 external_mini_cluster.cc:1428] Setting key 5685b9fcb05e0af7c7efe41951494215
I20260925 00:52:21.266374 19699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199: Bootstrap starting.
I20260925 00:52:21.268556 19699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:21.269290 19699 log.cc:824] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:21.271143 19699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199: No bootstrap required, opened a new log
I20260925 00:52:21.274806 19699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bf96dba5704d93a40c66d7f5a38199" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43035 } }
I20260925 00:52:21.274951 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.266590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.0.249.126:43035)
0925 00:52:21.267243 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:21.267258 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:21.267305 (+ 47us) server_negotiation.cc:407] Connection header received
0925 00:52:21.267967 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:21.267986 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:21.268377 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:21.268759 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:21.269231 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.270475 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.271521 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.271995 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.272189 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.273117 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:21.273122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:21.273134 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:21.273641 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:21.273711 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:21.273733 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:21.273853 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:21.273995 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 00:52:21.274247 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":116,"threads_started":1}
I20260925 00:52:21.275046 19699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:21.275116 19699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bf96dba5704d93a40c66d7f5a38199, State: Initialized, Role: FOLLOWER
I20260925 00:52:21.275595 19699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bf96dba5704d93a40c66d7f5a38199" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43035 } }
I20260925 00:52:21.275745 19699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:21.275805 19699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:21.275903 19699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:21.276714 19699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bf96dba5704d93a40c66d7f5a38199" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43035 } }
I20260925 00:52:21.277064 19699 leader_election.cc:302] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03bf96dba5704d93a40c66d7f5a38199; no voters:
I20260925 00:52:21.277369 19699 leader_election.cc:288] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:21.277477 19704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:21.277716 19704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [term 1 LEADER]: Becoming Leader. State: Replica: 03bf96dba5704d93a40c66d7f5a38199, State: Running, Role: LEADER
I20260925 00:52:21.278026 19704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bf96dba5704d93a40c66d7f5a38199" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43035 } }
I20260925 00:52:21.278652 19699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:21.279699 19706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bf96dba5704d93a40c66d7f5a38199. Latest consensus state: current_term: 1 leader_uuid: "03bf96dba5704d93a40c66d7f5a38199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bf96dba5704d93a40c66d7f5a38199" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43035 } } }
I20260925 00:52:21.279752 19705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03bf96dba5704d93a40c66d7f5a38199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bf96dba5704d93a40c66d7f5a38199" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 43035 } } }
I20260925 00:52:21.279924 19705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:21.279951 19706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:21.280658 19713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:21.283164 19713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:21.288146 19713 catalog_manager.cc:1347] Generated new cluster ID: 1b076485b43b40cda1f895f0a7959474
I20260925 00:52:21.288210 19713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:21.297879 19713 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:21.298727 19713 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:21.313755 19713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199: Generated new TSK 0
I20260925 00:52:21.314601 19713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:21.318866 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43035
--builtin_ntp_servers=127.0.249.84:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:52:21.431476 19723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:21.431782 19723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:21.431855 19723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:21.435866 19723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:21.435951 19723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:21.436055 19723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:52:21.440816 19723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:21.441929 19723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:21.443364 19723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:21.452632 19728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:21.452667 19729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:21.452737 19723 server_base.cc:1034] running on GCE node
W20260925 00:52:21.452667 19731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:21.453259 19723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:21.453908 19723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:21.455061 19723 hybrid_clock.cc:648] HybridClock initialized: now 1790297541455050 us; error 26 us; skew 500 ppm
I20260925 00:52:21.456871 19723 webserver.cc:460] Webserver started at http://127.0.249.65:35519/ using document root <none> and password file <none>
I20260925 00:52:21.457495 19723 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:21.457561 19723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:21.457767 19723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:21.459743 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "49dd994cce284fd09876f23ddf0d6552"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "09ef162ec9b07260d57fce89121a8fc0"
server_key_iv: "bea8298cabb98be56297089e4ee12653"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.460299 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "49dd994cce284fd09876f23ddf0d6552"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "09ef162ec9b07260d57fce89121a8fc0"
server_key_iv: "bea8298cabb98be56297089e4ee12653"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.463842 19723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:52:21.466190 19737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.467338 19723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:21.467469 19723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "49dd994cce284fd09876f23ddf0d6552"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "09ef162ec9b07260d57fce89121a8fc0"
server_key_iv: "bea8298cabb98be56297089e4ee12653"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.467576 19723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:21.486513 19723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:21.487167 19723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:21.487360 19723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:21.487910 19723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:21.488811 19723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:21.488883 19723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.488953 19723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:21.489006 19723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.497462 19723 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:45977
I20260925 00:52:21.497486 19850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:45977 every 8 connection(s)
I20260925 00:52:21.498441 19723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:52:21.505734 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 19723
I20260925 00:52:21.505857 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:52:21.506156 997 external_mini_cluster.cc:1428] Setting key 23c53c04e39a584aff55e4a33830a5ea
I20260925 00:52:21.508139 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.500284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:42163 (local address 127.0.249.126:43035)
0925 00:52:21.500422 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:21.500426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:21.501741 (+ 1315us) server_negotiation.cc:407] Connection header received
0925 00:52:21.503137 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:21.503139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:21.503187 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:21.503262 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:21.504126 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.504653 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.505897 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.506245 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.506352 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.507589 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:21.507592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:21.507613 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:21.507629 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:21.507652 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:21.507662 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:21.507734 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:21.507904 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 00:52:21.507962 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:52:21.508306 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43035
--builtin_ntp_servers=127.0.249.84:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:52:21.508996 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.500764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43035 (local address 127.0.249.65:42163)
0925 00:52:21.501561 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:21.501616 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:21.502924 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:21.503392 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:21.503398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:21.503481 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:21.503990 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.504003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.504759 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.504762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.505739 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.505749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.506388 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.506392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.506587 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.507313 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:52:21.507315 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:21.507354 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:52:21.507443 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:21.507753 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:21.507761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:21.507765 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 00:52:21.507873 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 00:52:21.508104 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":160,"threads_started":1}
I20260925 00:52:21.510475 19851 heartbeater.cc:347] Connected to a master server at 127.0.249.126:43035
I20260925 00:52:21.510699 19851 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:21.511188 19851 heartbeater.cc:511] Master 127.0.249.126:43035 requested a full tablet report, sending...
I20260925 00:52:21.512892 19663 ts_manager.cc:194] Registered new tserver with Master: 49dd994cce284fd09876f23ddf0d6552 (127.0.249.65:45977)
I20260925 00:52:21.514696 19663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.65:42163
W20260925 00:52:21.620222 19854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:21.620476 19854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:21.620525 19854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:21.624517 19854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:21.624581 19854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:21.624636 19854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:52:21.629272 19854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:21.630300 19854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:21.631596 19854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:21.640548 19859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:21.640658 19862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:21.640712 19860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:21.640904 19854 server_base.cc:1034] running on GCE node
I20260925 00:52:21.641301 19854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:21.641991 19854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:21.643155 19854 hybrid_clock.cc:648] HybridClock initialized: now 1790297541643128 us; error 33 us; skew 500 ppm
I20260925 00:52:21.644986 19854 webserver.cc:460] Webserver started at http://127.0.249.66:40669/ using document root <none> and password file <none>
I20260925 00:52:21.645641 19854 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:21.645709 19854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:21.645953 19854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:21.647909 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "d8d6f354dfe743199e5acb323abd74f8"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "f37c1166e3d67627f134db7f2d99a79b"
server_key_iv: "45a3679c6e172ef74812532e48e83985"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.648468 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "d8d6f354dfe743199e5acb323abd74f8"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "f37c1166e3d67627f134db7f2d99a79b"
server_key_iv: "45a3679c6e172ef74812532e48e83985"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.652050 19854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 00:52:21.654350 19868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.655463 19854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:21.655583 19854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "d8d6f354dfe743199e5acb323abd74f8"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "f37c1166e3d67627f134db7f2d99a79b"
server_key_iv: "45a3679c6e172ef74812532e48e83985"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.655685 19854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:21.693220 19854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:21.693953 19854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:21.694164 19854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:21.694763 19854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:21.695695 19854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:21.695742 19854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.695814 19854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:21.695858 19854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.705123 19854 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:36299
I20260925 00:52:21.705173 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:36299 every 8 connection(s)
I20260925 00:52:21.706065 19854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:52:21.715190 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 19854
I20260925 00:52:21.715306 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.707948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:55313 (local address 127.0.249.126:43035)
0925 00:52:21.708119 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:21.708124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:21.709127 (+ 1003us) server_negotiation.cc:407] Connection header received
0925 00:52:21.710596 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:21.710598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:21.710646 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:21.710727 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:21.711667 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.712178 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.713380 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.713642 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.713731 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.714789 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:21.714792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:21.714794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:21.714806 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:21.714824 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:21.714832 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:21.714890 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:21.715105 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 00:52:21.715155 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 00:52:21.715339 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:52:21.715683 997 external_mini_cluster.cc:1428] Setting key d9563b4cc9fc5c0ddb1ef15507b38db1
I20260925 00:52:21.716409 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.708269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43035 (local address 127.0.249.66:55313)
0925 00:52:21.708933 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:21.708977 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:21.710392 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:21.710898 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:21.710905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:21.711000 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:21.711526 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.711539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.712291 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.712294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.713268 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.713278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.713785 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.713788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.713938 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.714512 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:52:21.714514 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:21.714578 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:52:21.714642 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:21.714922 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:21.714933 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:21.714939 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 00:52:21.715106 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 00:52:21.715430 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260925 00:52:21.717615 19982 heartbeater.cc:347] Connected to a master server at 127.0.249.126:43035
I20260925 00:52:21.717877 19982 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:21.718062 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43035
--builtin_ntp_servers=127.0.249.84:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:52:21.718451 19982 heartbeater.cc:511] Master 127.0.249.126:43035 requested a full tablet report, sending...
I20260925 00:52:21.719442 19663 ts_manager.cc:194] Registered new tserver with Master: d8d6f354dfe743199e5acb323abd74f8 (127.0.249.66:36299)
I20260925 00:52:21.720100 19663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.66:55313
W20260925 00:52:21.832957 19985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:21.833204 19985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:21.833244 19985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:21.837332 19985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:21.837443 19985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:21.837527 19985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:52:21.842171 19985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:21.843060 19985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:21.844398 19985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:21.853770 19990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:21.853804 19985 server_base.cc:1034] running on GCE node
W20260925 00:52:21.853773 19993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:21.853772 19991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:21.854373 19985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:21.855032 19985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:21.856228 19985 hybrid_clock.cc:648] HybridClock initialized: now 1790297541856192 us; error 46 us; skew 500 ppm
I20260925 00:52:21.858111 19985 webserver.cc:460] Webserver started at http://127.0.249.67:38715/ using document root <none> and password file <none>
I20260925 00:52:21.858695 19985 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:21.858785 19985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:21.859051 19985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:21.861186 19985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "181954c50ed949c0b25a040f77e26a7f"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "1ba22db63152cb561315a43e80de20af"
server_key_iv: "704241a535e55baedfdc6e3a2f052aa5"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.861783 19985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "181954c50ed949c0b25a040f77e26a7f"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "1ba22db63152cb561315a43e80de20af"
server_key_iv: "704241a535e55baedfdc6e3a2f052aa5"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.865319 19985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:52:21.867650 19999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.868790 19985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:21.868928 19985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "181954c50ed949c0b25a040f77e26a7f"
format_stamp: "Formatted at 2026-09-25 00:52:21 on dist-test-slave-05m0"
server_key: "1ba22db63152cb561315a43e80de20af"
server_key_iv: "704241a535e55baedfdc6e3a2f052aa5"
server_key_version: "encryptionkey@0"
I20260925 00:52:21.869038 19985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:21.891474 19985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:21.892122 19985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:21.892324 19985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:21.892882 19985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:21.893855 19985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:21.893925 19985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.893992 19985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:21.894039 19985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:21.902530 19985 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:35253
I20260925 00:52:21.902616 20112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:35253 every 8 connection(s)
I20260925 00:52:21.903424 19985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:52:21.904369 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 19985
I20260925 00:52:21.904458 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:52:21.904744 997 external_mini_cluster.cc:1428] Setting key 3188079c1b78e17c393f8e14aaf40a85
I20260925 00:52:21.912201 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.905372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:38323 (local address 127.0.249.126:43035)
0925 00:52:21.905531 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:21.905534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:21.906733 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 00:52:21.908047 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:21.908049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:21.908099 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:21.908165 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:21.908932 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.909465 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.910513 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.910737 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.910816 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.911829 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:21.911832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:21.911833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:21.911843 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:21.911858 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:21.911865 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:21.911919 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:21.912036 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 00:52:21.912096 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 00:52:21.912955 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.905883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:43035 (local address 127.0.249.67:38323)
0925 00:52:21.906596 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:21.906644 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:21.907872 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:21.908290 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:21.908296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:21.908375 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:21.908822 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.908834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.909550 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.909553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.910409 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.910417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.910856 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.910859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.910996 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.911607 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:52:21.911609 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:21.911646 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:52:21.911710 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:21.911931 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:21.911935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:21.911938 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 00:52:21.912022 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 00:52:21.912224 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":160,"threads_started":1}
I20260925 00:52:21.914115 20113 heartbeater.cc:347] Connected to a master server at 127.0.249.126:43035
I20260925 00:52:21.914340 20113 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:21.914811 20113 heartbeater.cc:511] Master 127.0.249.126:43035 requested a full tablet report, sending...
I20260925 00:52:21.915732 19663 ts_manager.cc:194] Registered new tserver with Master: 181954c50ed949c0b25a040f77e26a7f (127.0.249.67:35253)
I20260925 00:52:21.916286 19663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.67:38323
I20260925 00:52:21.918426 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:52:21.924220 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.920255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.0.249.126:43035)
0925 00:52:21.920416 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:21.920420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:21.920620 (+ 200us) server_negotiation.cc:407] Connection header received
0925 00:52:21.920754 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:21.920757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:21.920804 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:21.920887 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:21.921176 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.921737 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.922566 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.922784 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.922887 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.923758 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:21.923762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:21.923767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:21.923783 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:21.923800 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:21.923808 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:21.923934 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:21.924051 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 00:52:21.924086 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 00:52:21.927081 19663 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:52:21.930028 19663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:52:21.944535 20127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.938519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:35253 (local address 127.0.0.1:40432)
0925 00:52:21.939573 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:21.939590 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:21.939686 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:21.940532 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:21.940535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:21.940555 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:21.940646 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.940653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.942030 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.942033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.943187 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.943195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.943995 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.943999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.944098 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.944133 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 00:52:21.944259 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 00:52:21.944410 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"thread_start_us":107,"threads_started":1}
I20260925 00:52:21.944542 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.939167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.0.249.67:35253)
0925 00:52:21.939976 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:21.939993 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:21.940015 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:52:21.940084 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:21.940089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:21.940234 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:21.940372 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:21.940781 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.941905 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.943356 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.943873 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.943995 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.944180 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:21.944283 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 00:52:21.944380 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":138,"threads_started":1}
I20260925 00:52:21.944830 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.938366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:36299 (local address 127.0.0.1:40898)
0925 00:52:21.938960 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:21.939005 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:21.939130 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:21.939929 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:21.939937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:21.939959 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:21.940080 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.940084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.941580 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.941583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.943066 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.943076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.944315 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.944319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.944498 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.944517 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 00:52:21.944590 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 00:52:21.944683 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":120,"thread_start_us":78,"threads_started":1}
I20260925 00:52:21.944926 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.939228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.0.249.65:45977)
0925 00:52:21.939719 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:21.939735 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:21.939877 (+ 142us) server_negotiation.cc:407] Connection header received
0925 00:52:21.940003 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:21.940010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:21.940194 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:21.940360 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:21.940534 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.941367 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.943035 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.943897 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.944041 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.944227 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:21.944671 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 00:52:21.944763 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":113,"threads_started":1}
I20260925 00:52:21.945156 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.938841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45977 (local address 127.0.0.1:51976)
0925 00:52:21.939758 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:21.939773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:21.939874 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:21.940344 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:21.940347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:21.940367 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:21.940440 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.940445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.941553 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.941559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.942894 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:21.942904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.944315 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.944319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:21.944481 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.944508 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 00:52:21.944590 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 00:52:21.944930 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":384,"thread_start_us":92,"threads_started":1}
I20260925 00:52:21.946367 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:21.938493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.0.249.66:36299)
0925 00:52:21.939416 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:21.939428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:21.939454 (+ 26us) server_negotiation.cc:407] Connection header received
0925 00:52:21.939536 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:21.939541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:21.939759 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:21.940203 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:21.940335 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.941454 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.943199 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:21.943971 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:21.944069 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:21.944227 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:21.946094 (+ 1867us) server_negotiation.cc:299] Negotiation successful
0925 00:52:21.946196 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":102,"threads_started":1}
I20260925 00:52:21.947580 20047 tablet_service.cc:1467] Processing CreateTablet for tablet 8b16926226b443a19de877646608dc65 (DEFAULT_TABLE table=test-table [id=d2c717481bc04bbea3ea3c098317ebda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:21.947610 19785 tablet_service.cc:1467] Processing CreateTablet for tablet 8b16926226b443a19de877646608dc65 (DEFAULT_TABLE table=test-table [id=d2c717481bc04bbea3ea3c098317ebda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:21.948529 19785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b16926226b443a19de877646608dc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:21.948549 20047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b16926226b443a19de877646608dc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:21.950376 19916 tablet_service.cc:1467] Processing CreateTablet for tablet 8b16926226b443a19de877646608dc65 (DEFAULT_TABLE table=test-table [id=d2c717481bc04bbea3ea3c098317ebda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:21.951294 19916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b16926226b443a19de877646608dc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:21.954946 20130 tablet_bootstrap.cc:492] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552: Bootstrap starting.
I20260925 00:52:21.957162 20130 tablet_bootstrap.cc:654] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:21.957719 20131 tablet_bootstrap.cc:492] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: Bootstrap starting.
I20260925 00:52:21.957796 20132 tablet_bootstrap.cc:492] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: Bootstrap starting.
I20260925 00:52:21.958122 20130 log.cc:824] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:21.959935 20131 tablet_bootstrap.cc:654] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:21.960013 20132 tablet_bootstrap.cc:654] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:21.960372 20130 tablet_bootstrap.cc:492] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552: No bootstrap required, opened a new log
I20260925 00:52:21.960614 20130 ts_tablet_manager.cc:1397] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:52:21.960798 20131 log.cc:824] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:21.960923 20132 log.cc:824] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:21.962602 20132 tablet_bootstrap.cc:492] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: No bootstrap required, opened a new log
I20260925 00:52:21.962803 20132 ts_tablet_manager.cc:1397] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:52:21.962836 20131 tablet_bootstrap.cc:492] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: No bootstrap required, opened a new log
I20260925 00:52:21.963018 20131 ts_tablet_manager.cc:1397] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:52:21.965118 20130 raft_consensus.cc:348] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } }
I20260925 00:52:21.965463 20130 raft_consensus.cc:374] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:21.965545 20130 raft_consensus.cc:729] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49dd994cce284fd09876f23ddf0d6552, State: Initialized, Role: FOLLOWER
I20260925 00:52:21.966001 20130 consensus_queue.cc:260] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } }
I20260925 00:52:21.965996 20132 raft_consensus.cc:348] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } }
I20260925 00:52:21.966204 20132 raft_consensus.cc:374] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:21.966272 20132 raft_consensus.cc:729] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d6f354dfe743199e5acb323abd74f8, State: Initialized, Role: FOLLOWER
I20260925 00:52:21.966727 19851 heartbeater.cc:503] Master 127.0.249.126:43035 was elected leader, sending a full tablet report...
I20260925 00:52:21.967113 20130 ts_tablet_manager.cc:1428] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:52:21.967293 20131 raft_consensus.cc:348] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } }
I20260925 00:52:21.967527 20131 raft_consensus.cc:374] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:21.967590 20131 raft_consensus.cc:729] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181954c50ed949c0b25a040f77e26a7f, State: Initialized, Role: FOLLOWER
I20260925 00:52:21.967535 20132 consensus_queue.cc:260] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } }
I20260925 00:52:21.967993 20131 consensus_queue.cc:260] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } }
I20260925 00:52:21.968250 19982 heartbeater.cc:503] Master 127.0.249.126:43035 was elected leader, sending a full tablet report...
I20260925 00:52:21.968621 20132 ts_tablet_manager.cc:1428] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 00:52:21.968937 20131 ts_tablet_manager.cc:1428] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 00:52:21.969180 20113 heartbeater.cc:503] Master 127.0.249.126:43035 was elected leader, sending a full tablet report...
W20260925 00:52:21.999718 19852 tablet.cc:2369] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:22.061688 20136 raft_consensus.cc:482] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:22.061901 20136 raft_consensus.cc:504] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } }
I20260925 00:52:22.063072 20136 leader_election.cc:288] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8d6f354dfe743199e5acb323abd74f8 (127.0.249.66:36299), 181954c50ed949c0b25a040f77e26a7f (127.0.249.67:35253)
I20260925 00:52:22.066942 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:22.063391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:60045 (local address 127.0.249.67:35253)
0925 00:52:22.063622 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:22.063627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:22.063773 (+ 146us) server_negotiation.cc:407] Connection header received
0925 00:52:22.063926 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:22.063931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:22.063973 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:22.064051 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:22.064349 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.064850 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.065998 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.066391 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.066515 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:22.066564 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:22.066746 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 00:52:22.066811 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260925 00:52:22.066983 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:22.063392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:49283 (local address 127.0.249.66:36299)
0925 00:52:22.063515 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:22.063520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:22.063686 (+ 166us) server_negotiation.cc:407] Connection header received
0925 00:52:22.063852 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:22.063857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:22.063907 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:22.063993 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:22.064304 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.064799 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.066012 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.066407 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.066533 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:22.066583 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:22.066794 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 00:52:22.066857 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:52:22.067005 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:22.063299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:36299 (local address 127.0.249.65:49283)
0925 00:52:22.063591 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:22.063603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:22.063731 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:22.064012 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:22.064014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:22.064029 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:22.064174 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:22.064179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.064910 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.064914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:22.065877 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:22.065885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.066528 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.066532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:22.066675 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:22.066692 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 00:52:22.066779 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 00:52:22.066845 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":56,"thread_start_us":85,"threads_started":1}
I20260925 00:52:22.067008 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:22.063299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:35253 (local address 127.0.249.65:60045)
0925 00:52:22.063685 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:22.063697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:22.063817 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:22.064071 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:22.064074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:22.064089 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:22.064229 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:22.064245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.064966 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.064969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:22.065866 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:22.065874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.066510 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.066514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:22.066628 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:22.066646 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 00:52:22.066732 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 00:52:22.066798 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":88,"thread_start_us":77,"threads_started":1}
I20260925 00:52:22.067729 20067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b16926226b443a19de877646608dc65" candidate_uuid: "49dd994cce284fd09876f23ddf0d6552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181954c50ed949c0b25a040f77e26a7f" is_pre_election: true
I20260925 00:52:22.067900 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b16926226b443a19de877646608dc65" candidate_uuid: "49dd994cce284fd09876f23ddf0d6552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d6f354dfe743199e5acb323abd74f8" is_pre_election: true
I20260925 00:52:22.068028 20067 raft_consensus.cc:2454] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49dd994cce284fd09876f23ddf0d6552 in term 0.
I20260925 00:52:22.068195 19936 raft_consensus.cc:2454] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49dd994cce284fd09876f23ddf0d6552 in term 0.
I20260925 00:52:22.068513 19739 leader_election.cc:302] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181954c50ed949c0b25a040f77e26a7f, 49dd994cce284fd09876f23ddf0d6552; no voters:
I20260925 00:52:22.068792 20136 raft_consensus.cc:2781] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:52:22.068881 20136 raft_consensus.cc:482] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:52:22.068936 20136 raft_consensus.cc:3037] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:22.069921 20136 raft_consensus.cc:504] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } }
I20260925 00:52:22.070314 20136 leader_election.cc:288] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [CANDIDATE]: Term 1 election: Requested vote from peers d8d6f354dfe743199e5acb323abd74f8 (127.0.249.66:36299), 181954c50ed949c0b25a040f77e26a7f (127.0.249.67:35253)
I20260925 00:52:22.070667 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b16926226b443a19de877646608dc65" candidate_uuid: "49dd994cce284fd09876f23ddf0d6552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d6f354dfe743199e5acb323abd74f8"
I20260925 00:52:22.070667 20067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b16926226b443a19de877646608dc65" candidate_uuid: "49dd994cce284fd09876f23ddf0d6552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181954c50ed949c0b25a040f77e26a7f"
I20260925 00:52:22.070784 20067 raft_consensus.cc:3037] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:22.070784 19936 raft_consensus.cc:3037] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:22.072053 19936 raft_consensus.cc:2454] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49dd994cce284fd09876f23ddf0d6552 in term 1.
I20260925 00:52:22.072053 20067 raft_consensus.cc:2454] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49dd994cce284fd09876f23ddf0d6552 in term 1.
I20260925 00:52:22.072345 19739 leader_election.cc:302] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181954c50ed949c0b25a040f77e26a7f, 49dd994cce284fd09876f23ddf0d6552; no voters:
I20260925 00:52:22.072508 20136 raft_consensus.cc:2781] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:22.072764 20136 raft_consensus.cc:686] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 1 LEADER]: Becoming Leader. State: Replica: 49dd994cce284fd09876f23ddf0d6552, State: Running, Role: LEADER
I20260925 00:52:22.073112 20136 consensus_queue.cc:237] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } }
I20260925 00:52:22.076121 19662 catalog_manager.cc:5495] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49dd994cce284fd09876f23ddf0d6552 (127.0.249.65). New cstate: current_term: 1 leader_uuid: "49dd994cce284fd09876f23ddf0d6552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49dd994cce284fd09876f23ddf0d6552" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 45977 } health_report { overall_health: HEALTHY } } }
I20260925 00:52:22.093577 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:22.090390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.0.249.65:45977)
0925 00:52:22.090520 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:22.090523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:22.090535 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:52:22.090675 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:22.090678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:22.090725 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:22.090807 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:22.091025 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.091490 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.092279 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:22.092474 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:22.092545 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:22.092817 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:22.093168 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:22.093268 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:22.093343 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 00:52:22.093370 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 00:52:22.098918 19936 raft_consensus.cc:1261] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [term 1 FOLLOWER]: Refusing update from remote peer 49dd994cce284fd09876f23ddf0d6552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:22.098918 20067 raft_consensus.cc:1261] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [term 1 FOLLOWER]: Refusing update from remote peer 49dd994cce284fd09876f23ddf0d6552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:22.099586 20136 consensus_queue.cc:1035] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181954c50ed949c0b25a040f77e26a7f" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:22.099749 20141 consensus_queue.cc:1035] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d6f354dfe743199e5acb323abd74f8" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:22.106827 20144 mvcc.cc:205] Tried to move back new op lower bound from 7333058732430360576 to 7333058732334075904. Current Snapshot: MvccSnapshot[applied={T|T < 7333058732430360576}]
I20260925 00:52:22.109711 20145 mvcc.cc:205] Tried to move back new op lower bound from 7333058732430360576 to 7333058732334075904. Current Snapshot: MvccSnapshot[applied={T|T < 7333058732430360576}]
I20260925 00:52:22.115258 19662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:22.115391 19662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:22.117769 19662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03bf96dba5704d93a40c66d7f5a38199: Sending DeleteTablet for 3 replicas of tablet 8b16926226b443a19de877646608dc65
I20260925 00:52:22.118527 20047 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b16926226b443a19de877646608dc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:22 UTC) from {username='slave'} at 127.0.0.1:40432
I20260925 00:52:22.118536 19916 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b16926226b443a19de877646608dc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:22 UTC) from {username='slave'} at 127.0.0.1:40898
I20260925 00:52:22.118803 19785 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b16926226b443a19de877646608dc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:22 UTC) from {username='slave'} at 127.0.0.1:51976
I20260925 00:52:22.119268 20155 tablet_replica.cc:330] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552: stopping tablet replica
I20260925 00:52:22.119338 20153 tablet_replica.cc:330] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: stopping tablet replica
I20260925 00:52:22.119588 20153 raft_consensus.cc:2229] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:52:22.119588 20155 raft_consensus.cc:2229] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:52:22.119851 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 19723
I20260925 00:52:22.119900 20155 raft_consensus.cc:2258] T 8b16926226b443a19de877646608dc65 P 49dd994cce284fd09876f23ddf0d6552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:22.119938 20153 raft_consensus.cc:2258] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:22.121459 20154 tablet_replica.cc:330] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: stopping tablet replica
I20260925 00:52:22.121707 20154 raft_consensus.cc:2229] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:52:22.121915 20154 raft_consensus.cc:2258] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:22.122704 20153 ts_tablet_manager.cc:1905] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:22.123062 20154 ts_tablet_manager.cc:1905] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:22.125331 20154 ts_tablet_manager.cc:1918] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:22.125538 20154 log.cc:1196] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/8b16926226b443a19de877646608dc65
I20260925 00:52:22.125903 20154 ts_tablet_manager.cc:1939] T 8b16926226b443a19de877646608dc65 P d8d6f354dfe743199e5acb323abd74f8: Deleting consensus metadata
I20260925 00:52:22.126096 20153 ts_tablet_manager.cc:1918] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:22.126199 20153 log.cc:1196] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/8b16926226b443a19de877646608dc65
I20260925 00:52:22.126506 20153 ts_tablet_manager.cc:1939] T 8b16926226b443a19de877646608dc65 P 181954c50ed949c0b25a040f77e26a7f: Deleting consensus metadata
I20260925 00:52:22.127120 19650 catalog_manager.cc:4841] TS d8d6f354dfe743199e5acb323abd74f8 (127.0.249.66:36299): tablet 8b16926226b443a19de877646608dc65 (table test-table [id=d2c717481bc04bbea3ea3c098317ebda]) successfully deleted
I20260925 00:52:22.127764 19648 catalog_manager.cc:4841] TS 181954c50ed949c0b25a040f77e26a7f (127.0.249.67:35253): tablet 8b16926226b443a19de877646608dc65 (table test-table [id=d2c717481bc04bbea3ea3c098317ebda]) successfully deleted
W20260925 00:52:22.128312 19647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:45977 (error 108)
I20260925 00:52:22.128371 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 19854
I20260925 00:52:22.133692 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 19985
I20260925 00:52:22.134761 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:22.133932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:45977 (local address 127.0.0.1:52004)
0925 00:52:22.134576 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:22.134658 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479}
W20260925 00:52:22.135277 19647 catalog_manager.cc:4584] TS 49dd994cce284fd09876f23ddf0d6552 (127.0.249.65:45977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b16926226b443a19de877646608dc65: Network error: Client connection negotiation failed: client connection to 127.0.249.65:45977: connect: Connection refused (error 111)
I20260925 00:52:22.139376 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 19632
2026-09-25T00:52:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1088 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:22 dist-test-slave-05m0 krb5kdc[20159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:22 dist-test-slave-05m0 krb5kdc[20159](info): set up 2 sockets
Sep 25 00:52:22 dist-test-slave-05m0 krb5kdc[20159](info): commencing operation
krb5kdc: starting...
W20260925 00:52:23.169324 997 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:23 dist-test-slave-05m0 krb5kdc[20159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:23.287999 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36085
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:36085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:23.399581 20172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:23.399891 20172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:23.399971 20172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:23.404026 20172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:23.404105 20172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:23.404142 20172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:23.404206 20172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:23.404259 20172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:23.409407 20172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:36085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:23.410449 20172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:23.411438 20172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:23.419876 20177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:23.419925 20180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:23.419966 20178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:23.420104 20172 server_base.cc:1034] running on GCE node
I20260925 00:52:23.420785 20172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:23.421967 20172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:23.423121 20172 hybrid_clock.cc:648] HybridClock initialized: now 1790297543423095 us; error 37 us; skew 500 ppm
Sep 25 00:52:23 dist-test-slave-05m0 krb5kdc[20159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:23.425922 20172 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:23.426753 20172 webserver.cc:460] Webserver started at http://127.0.249.126:37507/ using document root <none> and password file <none>
I20260925 00:52:23.427253 20172 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:23.427309 20172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:23.427469 20172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:23.429481 20172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "9eadfa022ca247c6bfaecec5819580e1"
format_stamp: "Formatted at 2026-09-25 00:52:23 on dist-test-slave-05m0"
server_key: "17198219a8154899e643f3a46a82dfe1"
server_key_iv: "00efc57cab8638d18e7342c88bc62f04"
server_key_version: "encryptionkey@0"
I20260925 00:52:23.430001 20172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "9eadfa022ca247c6bfaecec5819580e1"
format_stamp: "Formatted at 2026-09-25 00:52:23 on dist-test-slave-05m0"
server_key: "17198219a8154899e643f3a46a82dfe1"
server_key_iv: "00efc57cab8638d18e7342c88bc62f04"
server_key_version: "encryptionkey@0"
I20260925 00:52:23.433676 20172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 00:52:23.436085 20187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:23.437364 20172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:23.437554 20172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "9eadfa022ca247c6bfaecec5819580e1"
format_stamp: "Formatted at 2026-09-25 00:52:23 on dist-test-slave-05m0"
server_key: "17198219a8154899e643f3a46a82dfe1"
server_key_iv: "00efc57cab8638d18e7342c88bc62f04"
server_key_version: "encryptionkey@0"
I20260925 00:52:23.437690 20172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:23.454907 20172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:23.457918 20172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:23.458119 20172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:23.464990 20172 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:36085
I20260925 00:52:23.465075 20239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:36085 every 8 connection(s)
I20260925 00:52:23.466045 20172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:23.468698 20240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:23.473694 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 20172
I20260925 00:52:23.473835 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:23.474081 20240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1: Bootstrap starting.
I20260925 00:52:23.474125 997 external_mini_cluster.cc:1428] Setting key 3d33a833823f62b3cc69d98e40a8f5cb
I20260925 00:52:23.476536 20240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:23.477365 20240 log.cc:824] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:23.479389 20240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1: No bootstrap required, opened a new log
Sep 25 00:52:23 dist-test-slave-05m0 krb5kdc[20159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:23.483439 20240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eadfa022ca247c6bfaecec5819580e1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36085 } }
I20260925 00:52:23.483728 20240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:23.483816 20240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eadfa022ca247c6bfaecec5819580e1, State: Initialized, Role: FOLLOWER
I20260925 00:52:23.484289 20240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eadfa022ca247c6bfaecec5819580e1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36085 } }
I20260925 00:52:23.484421 20240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:23.484500 20240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:23.484604 20240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:23.485499 20240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eadfa022ca247c6bfaecec5819580e1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36085 } }
I20260925 00:52:23.485829 20240 leader_election.cc:302] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eadfa022ca247c6bfaecec5819580e1; no voters:
I20260925 00:52:23.486155 20240 leader_election.cc:288] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:23.486223 20245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:23.486421 20245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [term 1 LEADER]: Becoming Leader. State: Replica: 9eadfa022ca247c6bfaecec5819580e1, State: Running, Role: LEADER
I20260925 00:52:23.486773 20245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eadfa022ca247c6bfaecec5819580e1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36085 } }
I20260925 00:52:23.487432 20240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:23.488330 20247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eadfa022ca247c6bfaecec5819580e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eadfa022ca247c6bfaecec5819580e1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36085 } } }
I20260925 00:52:23.488734 20247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:23.488849 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:23.475383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.0.249.126:36085)
0925 00:52:23.475795 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:23.475809 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:23.475843 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:52:23.476446 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:23.476462 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:23.476828 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:23.477095 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:23.477997 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.479038 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.480193 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.480677 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.480861 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:23.483551 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:23.483556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:23.483568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:23.483639 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:23.485575 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:23.486008 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:23.486014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:23.486020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:23.486160 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:23.486495 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:23.486498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:23.486500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:23.486984 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:23.487205 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:23.487447 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 00:52:23.487764 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260925 00:52:23.488638 20246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eadfa022ca247c6bfaecec5819580e1. Latest consensus state: current_term: 1 leader_uuid: "9eadfa022ca247c6bfaecec5819580e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eadfa022ca247c6bfaecec5819580e1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36085 } } }
I20260925 00:52:23.489359 20246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1 [sys.catalog]: This master's current role is: LEADER
W20260925 00:52:23.492722 20260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:23.492807 20260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:23.492875 20254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:23.493636 20254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:23.499140 20254 catalog_manager.cc:1347] Generated new cluster ID: 49cb8b4427744c82ad68b643633a9ccf
I20260925 00:52:23.499224 20254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:23.509260 20254 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:23.510447 20254 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:23.518180 20254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1: Generated new TSK 0
I20260925 00:52:23.518965 20254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.249.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
Entry for principal kudu/127.0.249.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab.
I20260925 00:52:23.553129 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36085
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:23.667333 20266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:23.667562 20266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:23.667610 20266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:23.671500 20266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:23.671550 20266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:23.671609 20266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:52:23.676560 20266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.65.keytab
--principal=kudu/127.0.249.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:23.677490 20266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:23.678445 20266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:23.688272 20274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:23.688256 20266 server_base.cc:1034] running on GCE node
W20260925 00:52:23.688241 20272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:23.688248 20271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:23.688964 20266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:23.689752 20266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:23.690943 20266 hybrid_clock.cc:648] HybridClock initialized: now 1790297543690924 us; error 41 us; skew 500 ppm
Sep 25 00:52:23 dist-test-slave-05m0 krb5kdc[20159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:23.693632 20266 init.cc:377] Logged in from keytab as kudu/127.0.249.65@KRBTEST.COM (short username kudu)
I20260925 00:52:23.694445 20266 webserver.cc:460] Webserver started at http://127.0.249.65:33487/ using document root <none> and password file <none>
I20260925 00:52:23.694995 20266 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:23.695082 20266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:23.695328 20266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:23.697315 20266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "f06131422af7404e9876c9dbcbf030a9"
format_stamp: "Formatted at 2026-09-25 00:52:23 on dist-test-slave-05m0"
server_key: "377e5893f5d4a6786f88080026bc86f8"
server_key_iv: "3d843f8f057a3b98cd00857fd03afd81"
server_key_version: "encryptionkey@0"
I20260925 00:52:23.697921 20266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "f06131422af7404e9876c9dbcbf030a9"
format_stamp: "Formatted at 2026-09-25 00:52:23 on dist-test-slave-05m0"
server_key: "377e5893f5d4a6786f88080026bc86f8"
server_key_iv: "3d843f8f057a3b98cd00857fd03afd81"
server_key_version: "encryptionkey@0"
I20260925 00:52:23.701504 20266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:52:23.703855 20281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:23.705044 20266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:23.705188 20266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "f06131422af7404e9876c9dbcbf030a9"
format_stamp: "Formatted at 2026-09-25 00:52:23 on dist-test-slave-05m0"
server_key: "377e5893f5d4a6786f88080026bc86f8"
server_key_iv: "3d843f8f057a3b98cd00857fd03afd81"
server_key_version: "encryptionkey@0"
I20260925 00:52:23.705296 20266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:23.741879 20266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:23.744766 20266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:23.744971 20266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:23.745581 20266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:23.746412 20266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:23.746482 20266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:23.746553 20266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:23.746606 20266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:23.755287 20266 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:46583
I20260925 00:52:23.755309 20394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:46583 every 8 connection(s)
I20260925 00:52:23.756274 20266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:52:23.760531 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 20266
I20260925 00:52:23.760669 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:52:23.760957 997 external_mini_cluster.cc:1428] Setting key 1d5472b9dffe8c5245a2222a0c96acd2
Sep 25 00:52:23 dist-test-slave-05m0 krb5kdc[20159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.65@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:23.770569 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:23.757995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:43349 (local address 127.0.249.126:36085)
0925 00:52:23.758125 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:23.758129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:23.758928 (+ 799us) server_negotiation.cc:407] Connection header received
0925 00:52:23.760176 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:23.760179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:23.760222 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:23.760317 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:23.761778 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.762341 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.763440 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.763874 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.763978 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:23.766930 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:23.766933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:23.766935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:23.766957 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:23.768390 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:23.768971 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:23.768974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:23.768975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:23.769043 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:23.769491 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:23.769493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:23.769494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:23.769660 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:23.769737 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:23.770362 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 00:52:23.770454 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.249.66@KRBTEST.COM; defaulting to no policy
I20260925 00:52:23.771569 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:23.758373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36085 (local address 127.0.249.65:43349)
0925 00:52:23.758782 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:23.758824 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:23.759987 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:23.760534 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:23.760541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:23.761002 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:23.761624 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:23.761660 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.762461 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.762464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:23.763318 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:23.763327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.764052 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.764059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:23.764315 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:23.764923 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:23.764947 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:23.766759 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:23.768507 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:23.768515 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:23.768530 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:23.768871 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:23.769137 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:23.769141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:23.769145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:23.769342 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:23.769774 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:23.769781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:23.770096 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 00:52:23.770322 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 00:52:23.770654 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260925 00:52:23.772783 20395 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36085
I20260925 00:52:23.773053 20395 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.66@KRBTEST.COM" created.
I20260925 00:52:23.773643 20395 heartbeater.cc:511] Master 127.0.249.126:36085 requested a full tablet report, sending...
I20260925 00:52:23.775224 20204 ts_manager.cc:194] Registered new tserver with Master: f06131422af7404e9876c9dbcbf030a9 (127.0.249.65:46583)
I20260925 00:52:23.776295 20204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.65@KRBTEST.COM'} at 127.0.249.65:43349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
Entry for principal kudu/127.0.249.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab.
I20260925 00:52:23.788774 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36085
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:23.895797 20400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:23.896064 20400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:23.896136 20400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:23.900102 20400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:23.900189 20400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:23.900286 20400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:52:23.905333 20400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.66.keytab
--principal=kudu/127.0.249.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:23.906347 20400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:23.907322 20400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:23.916761 20408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:23.916775 20406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:23.916821 20405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:23.916903 20400 server_base.cc:1034] running on GCE node
I20260925 00:52:23.917426 20400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:23.918011 20400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:23.919165 20400 hybrid_clock.cc:648] HybridClock initialized: now 1790297543919141 us; error 35 us; skew 500 ppm
Sep 25 00:52:23 dist-test-slave-05m0 krb5kdc[20159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:23.921712 20400 init.cc:377] Logged in from keytab as kudu/127.0.249.66@KRBTEST.COM (short username kudu)
I20260925 00:52:23.922541 20400 webserver.cc:460] Webserver started at http://127.0.249.66:43087/ using document root <none> and password file <none>
I20260925 00:52:23.923075 20400 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:23.923166 20400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:23.923389 20400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:23.925334 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "181bb72a8f7545cda96a2f0310dd3b08"
format_stamp: "Formatted at 2026-09-25 00:52:23 on dist-test-slave-05m0"
server_key: "d57015a82321d2d3f0948d9b46ffbee4"
server_key_iv: "6463015e29a2871c1c4ed6c471dd6435"
server_key_version: "encryptionkey@0"
I20260925 00:52:23.926023 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "181bb72a8f7545cda96a2f0310dd3b08"
format_stamp: "Formatted at 2026-09-25 00:52:23 on dist-test-slave-05m0"
server_key: "d57015a82321d2d3f0948d9b46ffbee4"
server_key_iv: "6463015e29a2871c1c4ed6c471dd6435"
server_key_version: "encryptionkey@0"
I20260925 00:52:23.929625 20400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:52:23.931903 20415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:23.933005 20400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:23.933138 20400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "181bb72a8f7545cda96a2f0310dd3b08"
format_stamp: "Formatted at 2026-09-25 00:52:23 on dist-test-slave-05m0"
server_key: "d57015a82321d2d3f0948d9b46ffbee4"
server_key_iv: "6463015e29a2871c1c4ed6c471dd6435"
server_key_version: "encryptionkey@0"
I20260925 00:52:23.933245 20400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:23.952569 20400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:23.955452 20400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:23.955662 20400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:23.956250 20400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:23.957047 20400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:23.957130 20400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:23.957221 20400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:23.957273 20400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:23.966106 20400 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:36371
I20260925 00:52:23.966141 20528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:36371 every 8 connection(s)
I20260925 00:52:23.966939 20400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:52:23.974807 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 20400
I20260925 00:52:23.974961 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:52:23.975248 997 external_mini_cluster.cc:1428] Setting key ff5a3f82090bf8f9dabea7b16cd594ce
Sep 25 00:52:23 dist-test-slave-05m0 krb5kdc[20159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.66@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:23.980072 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:23.968547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:58961 (local address 127.0.249.126:36085)
0925 00:52:23.968682 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:23.968686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:23.969514 (+ 828us) server_negotiation.cc:407] Connection header received
0925 00:52:23.970790 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:23.970793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:23.970847 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:23.970955 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:23.972068 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.972626 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.973839 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.974059 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.974168 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:23.976617 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:23.976621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:23.976623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:23.976645 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:23.978026 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:23.978598 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:23.978604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:23.978609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:23.978692 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:23.979024 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:23.979028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:23.979030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:23.979211 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:23.979321 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:23.979793 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 00:52:23.979908 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:52:23.980845 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:23.968909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36085 (local address 127.0.249.66:58961)
0925 00:52:23.969337 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:23.969378 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:23.970587 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:23.971138 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:23.971144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:23.971479 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:23.971929 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:23.971941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.972749 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.972752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:23.973717 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:23.973726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:23.974213 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:23.974217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:23.974372 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:23.974966 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:23.974982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:23.976449 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:23.978157 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:23.978164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:23.978177 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:23.978496 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:23.978783 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:23.978787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:23.978789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:23.978928 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:23.979318 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:23.979324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:23.979560 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 00:52:23.979750 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 00:52:23.980048 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
I20260925 00:52:23.982203 20529 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36085
I20260925 00:52:23.982434 20529 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:23.982894 20529 heartbeater.cc:511] Master 127.0.249.126:36085 requested a full tablet report, sending...
I20260925 00:52:23.983969 20204 ts_manager.cc:194] Registered new tserver with Master: 181bb72a8f7545cda96a2f0310dd3b08 (127.0.249.66:36371)
WARNING: no policy specified for kudu/127.0.249.67@KRBTEST.COM; defaulting to no policy
I20260925 00:52:23.984814 20204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.66@KRBTEST.COM'} at 127.0.249.66:58961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
Entry for principal kudu/127.0.249.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab.
I20260925 00:52:24.003737 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36085
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:24.112957 20534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:24.113219 20534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:24.113260 20534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:24.117209 20534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:24.117264 20534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:24.117318 20534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:52:24.122529 20534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/krb5kdc/kudu_127.0.249.67.keytab
--principal=kudu/127.0.249.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:36085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:24.123548 20534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:24.124593 20534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:24.133886 20540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:24.133886 20542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:24.134080 20534 server_base.cc:1034] running on GCE node
W20260925 00:52:24.133953 20539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:24.134577 20534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:24.135231 20534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:24.136408 20534 hybrid_clock.cc:648] HybridClock initialized: now 1790297544136393 us; error 25 us; skew 500 ppm
Sep 25 00:52:24 dist-test-slave-05m0 krb5kdc[20159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:24.139125 20534 init.cc:377] Logged in from keytab as kudu/127.0.249.67@KRBTEST.COM (short username kudu)
I20260925 00:52:24.139947 20534 webserver.cc:460] Webserver started at http://127.0.249.67:37821/ using document root <none> and password file <none>
I20260925 00:52:24.140442 20534 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:24.140499 20534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:24.140654 20534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:24.142752 20534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "9da376cc7b534e26946c66dedd9549a1"
format_stamp: "Formatted at 2026-09-25 00:52:24 on dist-test-slave-05m0"
server_key: "12e387a228be7c7066a5559018801355"
server_key_iv: "b0cc92af5682e1428b876a420945cbab"
server_key_version: "encryptionkey@0"
I20260925 00:52:24.143290 20534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "9da376cc7b534e26946c66dedd9549a1"
format_stamp: "Formatted at 2026-09-25 00:52:24 on dist-test-slave-05m0"
server_key: "12e387a228be7c7066a5559018801355"
server_key_iv: "b0cc92af5682e1428b876a420945cbab"
server_key_version: "encryptionkey@0"
I20260925 00:52:24.146867 20534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:52:24.149204 20549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:24.150420 20534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:24.150537 20534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "9da376cc7b534e26946c66dedd9549a1"
format_stamp: "Formatted at 2026-09-25 00:52:24 on dist-test-slave-05m0"
server_key: "12e387a228be7c7066a5559018801355"
server_key_iv: "b0cc92af5682e1428b876a420945cbab"
server_key_version: "encryptionkey@0"
I20260925 00:52:24.150640 20534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:24.166769 20534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:24.170110 20534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:24.170312 20534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:24.170907 20534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:24.171715 20534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:24.171761 20534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:24.171828 20534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:24.171865 20534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:24.180311 20534 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:46483
I20260925 00:52:24.180341 20662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:46483 every 8 connection(s)
I20260925 00:52:24.181150 20534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:52:24.189796 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 20534
I20260925 00:52:24.189940 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:52:24.190234 997 external_mini_cluster.cc:1428] Setting key 38c9ad880294565a4c8f7fba32aa397f
Sep 25 00:52:24 dist-test-slave-05m0 krb5kdc[20159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.67@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:24.194857 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.182932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:43321 (local address 127.0.249.126:36085)
0925 00:52:24.183069 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.183073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.183843 (+ 770us) server_negotiation.cc:407] Connection header received
0925 00:52:24.185225 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.185229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.185285 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.185392 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:24.186521 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.187051 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.188205 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.188424 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.188539 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.191255 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:24.191258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:24.191261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:24.191286 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:24.192810 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:24.193302 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:24.193306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:24.193308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:24.193439 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:24.193788 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:24.193791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:24.193793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:24.193972 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:24.194082 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.194570 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.194696 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:52:24.195581 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.183293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:36085 (local address 127.0.249.67:43321)
0925 00:52:24.183703 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:24.183744 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:24.185013 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:24.185580 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:24.185587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:24.185917 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:24.186379 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.186391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.187176 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.187179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.188068 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.188077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.188586 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.188590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.188804 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.189322 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:24.189337 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:24.191109 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:24.192926 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:24.192931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:24.192945 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:24.193197 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:24.193537 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:24.193541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:24.193543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:24.193684 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:24.194099 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:24.194105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:24.194339 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 00:52:24.194517 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 00:52:24.194823 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260925 00:52:24.196581 20663 heartbeater.cc:347] Connected to a master server at 127.0.249.126:36085
I20260925 00:52:24.196815 20663 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:24.197285 20663 heartbeater.cc:511] Master 127.0.249.126:36085 requested a full tablet report, sending...
I20260925 00:52:24.198249 20204 ts_manager.cc:194] Registered new tserver with Master: 9da376cc7b534e26946c66dedd9549a1 (127.0.249.67:46483)
I20260925 00:52:24.198796 20204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.249.67@KRBTEST.COM'} at 127.0.249.67:43321
I20260925 00:52:24.203913 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:52:24.212713 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.205822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.0.249.126:36085)
0925 00:52:24.205933 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.205936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.206148 (+ 212us) server_negotiation.cc:407] Connection header received
0925 00:52:24.206239 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.206241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.206274 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.206338 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:24.206976 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.207457 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.208312 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.208508 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.208583 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.209907 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:24.209910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:24.209911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:24.209947 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:24.211406 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:24.211760 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:24.211762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:24.211764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:24.211832 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:24.212045 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:24.212047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:24.212048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:24.212236 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:24.212334 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.212516 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.212602 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:52:24.215613 20204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:52:24.218636 20204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:52:24.232388 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.227082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:46483 (local address 127.0.0.1:41730)
0925 00:52:24.227796 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:24.227810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:24.227891 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:24.228800 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:24.228803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:24.228819 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:24.228918 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.228923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.229999 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.230003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.231170 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.231180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.232000 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.232002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.232088 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.232117 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 00:52:24.232205 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 00:52:24.232284 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":97,"threads_started":1}
I20260925 00:52:24.232484 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.227604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.0.249.67:46483)
0925 00:52:24.228117 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.228128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.228150 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:52:24.228225 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.228232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.228446 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.228635 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:24.229034 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.229867 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.231290 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.231894 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.232005 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.232148 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.232253 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.232338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":165,"threads_started":1}
I20260925 00:52:24.233203 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.227536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:46583 (local address 127.0.0.1:54528)
0925 00:52:24.228296 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:24.228308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:24.228403 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:24.228922 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:24.228925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:24.228941 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:24.229032 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.229038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.230437 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.230440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.231755 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.231766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.232822 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.232826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.232913 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.232941 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 00:52:24.233013 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:52:24.233080 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":113,"thread_start_us":92,"threads_started":1}
I20260925 00:52:24.233695 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.227935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.0.249.65:46583)
0925 00:52:24.228218 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.228231 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.228398 (+ 167us) server_negotiation.cc:407] Connection header received
0925 00:52:24.228501 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.228511 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.228689 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.228830 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:24.229144 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.230308 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.231898 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.232731 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.232814 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.232900 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.233438 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.233539 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20260925 00:52:24.234238 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.226827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:36371 (local address 127.0.0.1:46806)
0925 00:52:24.227348 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:24.227393 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:24.227534 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:24.228622 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:24.228627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:24.228649 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:24.228792 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.228799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.230329 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.230335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.231755 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.231766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.233845 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.233849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.233939 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.233968 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 00:52:24.234035 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 00:52:24.234120 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":90,"thread_start_us":78,"threads_started":1}
I20260925 00:52:24.234853 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.226945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.0.249.66:36371)
0925 00:52:24.228090 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.228105 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.228127 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:52:24.228225 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.228232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.228456 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.228611 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:24.228910 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.230177 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.232947 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.233707 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.234518 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.234640 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.234679 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.234741 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":124,"threads_started":1}
I20260925 00:52:24.235529 20597 tablet_service.cc:1467] Processing CreateTablet for tablet 17ed54e4bdcd42e787b1e566eda20e2e (DEFAULT_TABLE table=test-table [id=f4b25066ef3446a2aab160636e7fe816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:24.236501 20597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ed54e4bdcd42e787b1e566eda20e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:24.237339 20463 tablet_service.cc:1467] Processing CreateTablet for tablet 17ed54e4bdcd42e787b1e566eda20e2e (DEFAULT_TABLE table=test-table [id=f4b25066ef3446a2aab160636e7fe816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:24.238396 20463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ed54e4bdcd42e787b1e566eda20e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:24.237982 20329 tablet_service.cc:1467] Processing CreateTablet for tablet 17ed54e4bdcd42e787b1e566eda20e2e (DEFAULT_TABLE table=test-table [id=f4b25066ef3446a2aab160636e7fe816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:24.238967 20329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ed54e4bdcd42e787b1e566eda20e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:24.242772 20680 tablet_bootstrap.cc:492] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: Bootstrap starting.
I20260925 00:52:24.244792 20680 tablet_bootstrap.cc:654] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:24.245362 20682 tablet_bootstrap.cc:492] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9: Bootstrap starting.
I20260925 00:52:24.246138 20680 log.cc:824] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:24.246258 20681 tablet_bootstrap.cc:492] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08: Bootstrap starting.
I20260925 00:52:24.247658 20682 tablet_bootstrap.cc:654] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:24.248410 20681 tablet_bootstrap.cc:654] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:24.248692 20682 log.cc:824] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:24.249173 20681 log.cc:824] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:24.250734 20682 tablet_bootstrap.cc:492] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9: No bootstrap required, opened a new log
I20260925 00:52:24.250747 20680 tablet_bootstrap.cc:492] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: No bootstrap required, opened a new log
I20260925 00:52:24.250782 20681 tablet_bootstrap.cc:492] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08: No bootstrap required, opened a new log
I20260925 00:52:24.250929 20682 ts_tablet_manager.cc:1397] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 00:52:24.250928 20680 ts_tablet_manager.cc:1397] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260925 00:52:24.250988 20681 ts_tablet_manager.cc:1397] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:52:24.254254 20681 raft_consensus.cc:348] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } }
I20260925 00:52:24.254560 20681 raft_consensus.cc:374] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:24.254617 20681 raft_consensus.cc:729] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181bb72a8f7545cda96a2f0310dd3b08, State: Initialized, Role: FOLLOWER
I20260925 00:52:24.254793 20682 raft_consensus.cc:348] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } }
I20260925 00:52:24.254793 20680 raft_consensus.cc:348] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } }
I20260925 00:52:24.255033 20680 raft_consensus.cc:374] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:24.255033 20682 raft_consensus.cc:374] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:24.255092 20680 raft_consensus.cc:729] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da376cc7b534e26946c66dedd9549a1, State: Initialized, Role: FOLLOWER
I20260925 00:52:24.255067 20681 consensus_queue.cc:260] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } }
I20260925 00:52:24.255151 20682 raft_consensus.cc:729] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06131422af7404e9876c9dbcbf030a9, State: Initialized, Role: FOLLOWER
I20260925 00:52:24.255517 20680 consensus_queue.cc:260] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } }
I20260925 00:52:24.255517 20682 consensus_queue.cc:260] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } }
I20260925 00:52:24.255870 20529 heartbeater.cc:503] Master 127.0.249.126:36085 was elected leader, sending a full tablet report...
I20260925 00:52:24.256456 20680 ts_tablet_manager.cc:1428] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260925 00:52:24.256456 20682 ts_tablet_manager.cc:1428] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 00:52:24.256801 20663 heartbeater.cc:503] Master 127.0.249.126:36085 was elected leader, sending a full tablet report...
I20260925 00:52:24.257184 20681 ts_tablet_manager.cc:1428] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 00:52:24.257306 20396 tablet.cc:2369] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:24.257571 20395 heartbeater.cc:503] Master 127.0.249.126:36085 was elected leader, sending a full tablet report...
I20260925 00:52:24.344879 20687 raft_consensus.cc:482] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:24.345068 20687 raft_consensus.cc:504] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } }
I20260925 00:52:24.346285 20687 leader_election.cc:288] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 181bb72a8f7545cda96a2f0310dd3b08 (127.0.249.66:36371), f06131422af7404e9876c9dbcbf030a9 (127.0.249.65:46583)
I20260925 00:52:24.350602 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.346524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:36371 (local address 127.0.249.67:46639)
0925 00:52:24.346646 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:24.346662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:24.346764 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:24.347138 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:24.347141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:24.347168 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:24.347264 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.347268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.348227 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.348230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.349339 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.349349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.350165 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.350169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.350276 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.350301 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 00:52:24.350390 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 00:52:24.350464 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 00:52:24.350728 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.346686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:46639 (local address 127.0.249.66:36371)
0925 00:52:24.346818 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.346824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.346843 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:52:24.346923 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.346926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.346996 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.347110 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:24.347405 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.348111 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.349523 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.350046 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.350147 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.350189 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.350435 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.350562 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:52:24.350900 20689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.346816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:46583 (local address 127.0.249.67:55733)
0925 00:52:24.347203 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:24.347220 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:24.347330 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:24.347583 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:24.347586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:24.347606 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:24.347737 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.347744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.348568 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.348571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.349711 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.349722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.350453 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.350457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.350564 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.350590 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 00:52:24.350680 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 00:52:24.350750 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":118,"threads_started":1}
I20260925 00:52:24.351254 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.346951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:55733 (local address 127.0.249.65:46583)
0925 00:52:24.347098 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.347104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.347320 (+ 216us) server_negotiation.cc:407] Connection header received
0925 00:52:24.347443 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.347445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.347484 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.347539 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:24.347849 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.348441 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.349840 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.350327 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.350435 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.350478 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.351043 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.351122 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 00:52:24.351411 20483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ed54e4bdcd42e787b1e566eda20e2e" candidate_uuid: "9da376cc7b534e26946c66dedd9549a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181bb72a8f7545cda96a2f0310dd3b08" is_pre_election: true
I20260925 00:52:24.351759 20483 raft_consensus.cc:2454] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da376cc7b534e26946c66dedd9549a1 in term 0.
I20260925 00:52:24.351794 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ed54e4bdcd42e787b1e566eda20e2e" candidate_uuid: "9da376cc7b534e26946c66dedd9549a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06131422af7404e9876c9dbcbf030a9" is_pre_election: true
I20260925 00:52:24.352092 20349 raft_consensus.cc:2454] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da376cc7b534e26946c66dedd9549a1 in term 0.
I20260925 00:52:24.352322 20552 leader_election.cc:302] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181bb72a8f7545cda96a2f0310dd3b08, 9da376cc7b534e26946c66dedd9549a1; no voters:
I20260925 00:52:24.352604 20687 raft_consensus.cc:2781] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:52:24.352681 20687 raft_consensus.cc:482] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:52:24.352711 20687 raft_consensus.cc:3037] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:24.353823 20687 raft_consensus.cc:504] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } }
I20260925 00:52:24.354152 20687 leader_election.cc:288] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [CANDIDATE]: Term 1 election: Requested vote from peers 181bb72a8f7545cda96a2f0310dd3b08 (127.0.249.66:36371), f06131422af7404e9876c9dbcbf030a9 (127.0.249.65:46583)
I20260925 00:52:24.354573 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ed54e4bdcd42e787b1e566eda20e2e" candidate_uuid: "9da376cc7b534e26946c66dedd9549a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06131422af7404e9876c9dbcbf030a9"
I20260925 00:52:24.354638 20483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ed54e4bdcd42e787b1e566eda20e2e" candidate_uuid: "9da376cc7b534e26946c66dedd9549a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181bb72a8f7545cda96a2f0310dd3b08"
I20260925 00:52:24.354724 20349 raft_consensus.cc:3037] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:24.354777 20483 raft_consensus.cc:3037] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:24.355759 20349 raft_consensus.cc:2454] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da376cc7b534e26946c66dedd9549a1 in term 1.
I20260925 00:52:24.355927 20483 raft_consensus.cc:2454] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da376cc7b534e26946c66dedd9549a1 in term 1.
I20260925 00:52:24.356065 20552 leader_election.cc:302] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da376cc7b534e26946c66dedd9549a1, f06131422af7404e9876c9dbcbf030a9; no voters:
I20260925 00:52:24.356266 20687 raft_consensus.cc:2781] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:24.356482 20687 raft_consensus.cc:686] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 1 LEADER]: Becoming Leader. State: Replica: 9da376cc7b534e26946c66dedd9549a1, State: Running, Role: LEADER
I20260925 00:52:24.356801 20687 consensus_queue.cc:237] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } }
I20260925 00:52:24.359911 20202 catalog_manager.cc:5495] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9da376cc7b534e26946c66dedd9549a1 (127.0.249.67). New cstate: current_term: 1 leader_uuid: "9da376cc7b534e26946c66dedd9549a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da376cc7b534e26946c66dedd9549a1" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 46483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 } health_report { overall_health: UNKNOWN } } }
I20260925 00:52:24.387106 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.383378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.0.249.67:46483)
0925 00:52:24.383521 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.383526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.383541 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:52:24.383690 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.383696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.383757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.383870 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:24.384115 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.384626 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.385647 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.385877 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.385988 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.386297 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:24.386706 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:24.386840 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.386881 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.386934 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 00:52:24.393286 20349 raft_consensus.cc:1261] T 17ed54e4bdcd42e787b1e566eda20e2e P f06131422af7404e9876c9dbcbf030a9 [term 1 FOLLOWER]: Refusing update from remote peer 9da376cc7b534e26946c66dedd9549a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:24.393494 20483 raft_consensus.cc:1261] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [term 1 FOLLOWER]: Refusing update from remote peer 9da376cc7b534e26946c66dedd9549a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:24.394054 20687 consensus_queue.cc:1035] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06131422af7404e9876c9dbcbf030a9" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:24.394300 20690 consensus_queue.cc:1035] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181bb72a8f7545cda96a2f0310dd3b08" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 36371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:24.404485 20694 mvcc.cc:205] Tried to move back new op lower bound from 7333058741826273280 to 7333058741688446976. Current Snapshot: MvccSnapshot[applied={T|T < 7333058741826273280}]
I20260925 00:52:24.406097 20695 mvcc.cc:205] Tried to move back new op lower bound from 7333058741826273280 to 7333058741688446976. Current Snapshot: MvccSnapshot[applied={T|T < 7333058741826273280}]
I20260925 00:52:24.409175 20203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:24.409384 20203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:24.411635 20203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9eadfa022ca247c6bfaecec5819580e1: Sending DeleteTablet for 3 replicas of tablet 17ed54e4bdcd42e787b1e566eda20e2e
I20260925 00:52:24.412385 20597 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ed54e4bdcd42e787b1e566eda20e2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:24 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:41730
I20260925 00:52:24.412389 20329 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ed54e4bdcd42e787b1e566eda20e2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:24 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:54528
I20260925 00:52:24.412389 20463 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ed54e4bdcd42e787b1e566eda20e2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:24 UTC) from {username='kudu', principal='kudu/127.0.249.126@KRBTEST.COM'} at 127.0.0.1:46806
I20260925 00:52:24.412837 20703 tablet_replica.cc:330] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08: stopping tablet replica
I20260925 00:52:24.413147 20703 raft_consensus.cc:2229] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:52:24.413436 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 20266
I20260925 00:52:24.413457 20703 raft_consensus.cc:2258] T 17ed54e4bdcd42e787b1e566eda20e2e P 181bb72a8f7545cda96a2f0310dd3b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:24.414842 20702 tablet_replica.cc:330] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: stopping tablet replica
I20260925 00:52:24.415100 20702 raft_consensus.cc:2229] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:52:24.415411 20702 raft_consensus.cc:2258] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:24.416860 20702 ts_tablet_manager.cc:1905] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:24.419636 20702 ts_tablet_manager.cc:1918] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 00:52:24.419648 20190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:46583 (error 108)
I20260925 00:52:24.419730 20702 log.cc:1196] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/17ed54e4bdcd42e787b1e566eda20e2e
I20260925 00:52:24.420061 20702 ts_tablet_manager.cc:1939] T 17ed54e4bdcd42e787b1e566eda20e2e P 9da376cc7b534e26946c66dedd9549a1: Deleting consensus metadata
I20260925 00:52:24.420051 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 20400
I20260925 00:52:24.421341 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.421006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:46583 (local address 127.0.0.1:54542)
0925 00:52:24.421152 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:24.421242 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 00:52:24.421790 20190 catalog_manager.cc:4584] TS f06131422af7404e9876c9dbcbf030a9 (127.0.249.65:46583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17ed54e4bdcd42e787b1e566eda20e2e: Network error: Client connection negotiation failed: client connection to 127.0.249.65:46583: connect: Connection refused (error 111)
I20260925 00:52:24.423002 20188 catalog_manager.cc:4841] TS 9da376cc7b534e26946c66dedd9549a1 (127.0.249.67:46483): tablet 17ed54e4bdcd42e787b1e566eda20e2e (table test-table [id=f4b25066ef3446a2aab160636e7fe816]) successfully deleted
I20260925 00:52:24.425885 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 20534
I20260925 00:52:24.426550 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.426018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:36371 (local address 127.0.0.1:46818)
0925 00:52:24.426431 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:24.426478 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.66:36371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349}
I20260925 00:52:24.431490 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 20172
2026-09-25T00:52:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2294 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T00:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:24Z Disabled control of system clock
I20260925 00:52:24.465022 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:40627
--webserver_interface=127.0.249.126
--webserver_port=0
--builtin_ntp_servers=127.0.249.84:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:40627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:52:24.577754 20709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:24.578006 20709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:24.578074 20709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:24.582072 20709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:24.582126 20709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:24.582141 20709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:24.582190 20709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:24.582218 20709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:24.587052 20709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:40627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:40627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:24.588063 20709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:24.589473 20709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:24.597649 20715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:24.597738 20717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:24.598006 20709 server_base.cc:1034] running on GCE node
W20260925 00:52:24.597649 20714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:24.598562 20709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:24.599691 20709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:24.600880 20709 hybrid_clock.cc:648] HybridClock initialized: now 1790297544600853 us; error 56 us; skew 500 ppm
I20260925 00:52:24.602713 20709 webserver.cc:460] Webserver started at http://127.0.249.126:33145/ using document root <none> and password file <none>
I20260925 00:52:24.603281 20709 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:24.603374 20709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:24.603601 20709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:24.605720 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "451b33badfff48c28dfc01172221488d"
format_stamp: "Formatted at 2026-09-25 00:52:24 on dist-test-slave-05m0"
server_key: "d15625a6886fed3a3b9632c26788e996"
server_key_iv: "f3cfff8af923866244e7ca9c4120545e"
server_key_version: "encryptionkey@0"
I20260925 00:52:24.606293 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "451b33badfff48c28dfc01172221488d"
format_stamp: "Formatted at 2026-09-25 00:52:24 on dist-test-slave-05m0"
server_key: "d15625a6886fed3a3b9632c26788e996"
server_key_iv: "f3cfff8af923866244e7ca9c4120545e"
server_key_version: "encryptionkey@0"
I20260925 00:52:24.609882 20709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 00:52:24.612197 20723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:24.613359 20709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:24.613543 20709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "451b33badfff48c28dfc01172221488d"
format_stamp: "Formatted at 2026-09-25 00:52:24 on dist-test-slave-05m0"
server_key: "d15625a6886fed3a3b9632c26788e996"
server_key_iv: "f3cfff8af923866244e7ca9c4120545e"
server_key_version: "encryptionkey@0"
I20260925 00:52:24.613655 20709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:24.627571 20709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:24.628309 20709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:24.628504 20709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:24.635362 20709 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:40627
I20260925 00:52:24.635386 20775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:40627 every 8 connection(s)
I20260925 00:52:24.636312 20709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:24.639356 20776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:24.640811 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 20709
I20260925 00:52:24.640944 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:24.641247 997 external_mini_cluster.cc:1428] Setting key fb7c0f8ca245c71011bc18e84da2c3bc
I20260925 00:52:24.645678 20776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d: Bootstrap starting.
I20260925 00:52:24.647778 20776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:24.648613 20776 log.cc:824] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:24.650640 20776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d: No bootstrap required, opened a new log
I20260925 00:52:24.650640 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.642746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.0.249.126:40627)
0925 00:52:24.643394 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.643409 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.643444 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:52:24.644092 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.644109 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.644503 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.644861 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:24.645281 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.646363 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.647241 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.647716 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.647883 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.648783 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:24.648790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:24.648813 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:24.649317 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:24.649408 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:24.649435 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:24.649554 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.649707 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.649967 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":132,"threads_started":1}
I20260925 00:52:24.654469 20776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451b33badfff48c28dfc01172221488d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40627 } }
I20260925 00:52:24.654714 20776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:24.654793 20776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451b33badfff48c28dfc01172221488d, State: Initialized, Role: FOLLOWER
I20260925 00:52:24.655215 20776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451b33badfff48c28dfc01172221488d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40627 } }
I20260925 00:52:24.655365 20776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:24.655452 20776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:24.655570 20776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:24.656512 20776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451b33badfff48c28dfc01172221488d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40627 } }
I20260925 00:52:24.656881 20776 leader_election.cc:302] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451b33badfff48c28dfc01172221488d; no voters:
I20260925 00:52:24.657220 20776 leader_election.cc:288] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:24.657387 20781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:24.657739 20781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [term 1 LEADER]: Becoming Leader. State: Replica: 451b33badfff48c28dfc01172221488d, State: Running, Role: LEADER
I20260925 00:52:24.658051 20781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451b33badfff48c28dfc01172221488d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40627 } }
I20260925 00:52:24.658502 20776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:24.659706 20783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451b33badfff48c28dfc01172221488d. Latest consensus state: current_term: 1 leader_uuid: "451b33badfff48c28dfc01172221488d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451b33badfff48c28dfc01172221488d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40627 } } }
I20260925 00:52:24.659829 20783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:24.660126 20782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "451b33badfff48c28dfc01172221488d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451b33badfff48c28dfc01172221488d" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 40627 } } }
I20260925 00:52:24.660250 20782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:24.660235 20790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:24.663498 20790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:24.669180 20790 catalog_manager.cc:1347] Generated new cluster ID: 6f8c0ab4ed8d4930abd2f0f38616be36
I20260925 00:52:24.669253 20790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:24.679605 20790 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:24.680583 20790 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:24.685431 20790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d: Generated new TSK 0
I20260925 00:52:24.686230 20790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:24.689519 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.65:0
--local_ip_for_outbound_sockets=127.0.249.65
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40627
--builtin_ntp_servers=127.0.249.84:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 00:52:24.804239 20800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:24.804508 20800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:24.804553 20800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:24.808988 20800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:24.809051 20800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:24.809134 20800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.65
I20260925 00:52:24.813849 20800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.249.65
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.65
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:24.814862 20800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:24.816347 20800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:24.825747 20806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:24.825762 20808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:24.825739 20805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:24.826056 20800 server_base.cc:1034] running on GCE node
I20260925 00:52:24.826478 20800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:24.827080 20800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:24.828264 20800 hybrid_clock.cc:648] HybridClock initialized: now 1790297544828251 us; error 24 us; skew 500 ppm
I20260925 00:52:24.830071 20800 webserver.cc:460] Webserver started at http://127.0.249.65:39973/ using document root <none> and password file <none>
I20260925 00:52:24.830629 20800 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:24.830721 20800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:24.830950 20800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:24.832885 20800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/data/instance:
uuid: "0feec57afb5a4cf59353dd5a4d79a435"
format_stamp: "Formatted at 2026-09-25 00:52:24 on dist-test-slave-05m0"
server_key: "dd6d53bb46c7e9e13afc8610cdeba9c5"
server_key_iv: "2ceaea00721f1576eb99c96dbacc274e"
server_key_version: "encryptionkey@0"
I20260925 00:52:24.833508 20800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/wal/instance:
uuid: "0feec57afb5a4cf59353dd5a4d79a435"
format_stamp: "Formatted at 2026-09-25 00:52:24 on dist-test-slave-05m0"
server_key: "dd6d53bb46c7e9e13afc8610cdeba9c5"
server_key_iv: "2ceaea00721f1576eb99c96dbacc274e"
server_key_version: "encryptionkey@0"
I20260925 00:52:24.837038 20800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 00:52:24.839468 20814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:24.840663 20800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 00:52:24.840796 20800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/wal
uuid: "0feec57afb5a4cf59353dd5a4d79a435"
format_stamp: "Formatted at 2026-09-25 00:52:24 on dist-test-slave-05m0"
server_key: "dd6d53bb46c7e9e13afc8610cdeba9c5"
server_key_iv: "2ceaea00721f1576eb99c96dbacc274e"
server_key_version: "encryptionkey@0"
I20260925 00:52:24.840905 20800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:24.864087 20800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:24.864794 20800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:24.864987 20800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:24.865639 20800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:24.866550 20800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:24.866596 20800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:24.866636 20800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:24.866698 20800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:24.875191 20800 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.65:44589
I20260925 00:52:24.875278 20927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.65:44589 every 8 connection(s)
I20260925 00:52:24.876137 20800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/data/info.pb
I20260925 00:52:24.885375 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.877977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.65:60827 (local address 127.0.249.126:40627)
0925 00:52:24.878149 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:24.878153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:24.879186 (+ 1033us) server_negotiation.cc:407] Connection header received
0925 00:52:24.880599 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:24.880603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:24.880665 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:24.880778 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:24.881732 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.882277 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.883415 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.883734 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.883849 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.884867 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:24.884871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:24.884874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:24.884886 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:24.884907 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:24.884922 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:24.884997 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:24.885166 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 00:52:24.885230 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:52:24.885866 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 20800
I20260925 00:52:24.885988 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-0/wal/instance
I20260925 00:52:24.886323 997 external_mini_cluster.cc:1428] Setting key f74779916cedc3cb10d6ac3ae7c183ef
I20260925 00:52:24.886406 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:24.878388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:40627 (local address 127.0.249.65:60827)
0925 00:52:24.879029 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:24.879067 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:24.880381 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:24.880955 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:24.880963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:24.881046 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:24.881579 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.881600 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.882406 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.882409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.883288 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:24.883297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:24.883886 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:24.883889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:24.884051 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:24.884614 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:52:24.884616 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:24.884654 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:52:24.884715 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:24.885011 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:24.885022 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:24.885029 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 00:52:24.885159 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 00:52:24.885445 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260925 00:52:24.887668 20928 heartbeater.cc:347] Connected to a master server at 127.0.249.126:40627
I20260925 00:52:24.887949 20928 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:24.888411 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.66:0
--local_ip_for_outbound_sockets=127.0.249.66
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40627
--builtin_ntp_servers=127.0.249.84:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:52:24.888626 20928 heartbeater.cc:511] Master 127.0.249.126:40627 requested a full tablet report, sending...
I20260925 00:52:24.890213 20739 ts_manager.cc:194] Registered new tserver with Master: 0feec57afb5a4cf59353dd5a4d79a435 (127.0.249.65:44589)
I20260925 00:52:24.891393 20739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.65:60827
W20260925 00:52:24.999053 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:24.999302 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:24.999341 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:25.003415 20931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:25.003473 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:25.003573 20931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.66
I20260925 00:52:25.008325 20931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.249.66
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.66
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:25.009361 20931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:25.010758 20931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:25.020241 20937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:25.020231 20936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:25.020211 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:25.020445 20931 server_base.cc:1034] running on GCE node
I20260925 00:52:25.020953 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:25.021649 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:25.022827 20931 hybrid_clock.cc:648] HybridClock initialized: now 1790297545022802 us; error 41 us; skew 500 ppm
I20260925 00:52:25.024559 20931 webserver.cc:460] Webserver started at http://127.0.249.66:37477/ using document root <none> and password file <none>
I20260925 00:52:25.025136 20931 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:25.025226 20931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:25.025507 20931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:25.027469 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/data/instance:
uuid: "16bffc3097e847499f2801c8fbdfe71e"
format_stamp: "Formatted at 2026-09-25 00:52:25 on dist-test-slave-05m0"
server_key: "3cc79cbc10ba60850e63679d7a754ba5"
server_key_iv: "bae24badc45184d4545ab5fb31e57106"
server_key_version: "encryptionkey@0"
I20260925 00:52:25.028029 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/wal/instance:
uuid: "16bffc3097e847499f2801c8fbdfe71e"
format_stamp: "Formatted at 2026-09-25 00:52:25 on dist-test-slave-05m0"
server_key: "3cc79cbc10ba60850e63679d7a754ba5"
server_key_iv: "bae24badc45184d4545ab5fb31e57106"
server_key_version: "encryptionkey@0"
I20260925 00:52:25.031682 20931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 00:52:25.033982 20945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:25.035140 20931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:25.035259 20931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/wal
uuid: "16bffc3097e847499f2801c8fbdfe71e"
format_stamp: "Formatted at 2026-09-25 00:52:25 on dist-test-slave-05m0"
server_key: "3cc79cbc10ba60850e63679d7a754ba5"
server_key_iv: "bae24badc45184d4545ab5fb31e57106"
server_key_version: "encryptionkey@0"
I20260925 00:52:25.035357 20931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:25.044108 20931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:25.044780 20931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:25.044914 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:25.045373 20931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:25.046255 20931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:25.046300 20931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:25.046336 20931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:25.046357 20931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:25.054817 20931 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.66:44859
I20260925 00:52:25.054847 21058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.66:44859 every 8 connection(s)
I20260925 00:52:25.055660 20931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/data/info.pb
I20260925 00:52:25.064692 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 20931
I20260925 00:52:25.064720 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.057313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:58489 (local address 127.0.249.126:40627)
0925 00:52:25.057553 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:25.057557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:25.058536 (+ 979us) server_negotiation.cc:407] Connection header received
0925 00:52:25.060119 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:25.060123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:25.060182 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:25.060262 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:25.061138 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.061760 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.062882 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.063100 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.063208 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.064198 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:25.064201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:25.064204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:25.064223 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:25.064241 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:25.064258 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:25.064336 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:25.064513 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 00:52:25.064578 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:52:25.064838 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/wal/instance
I20260925 00:52:25.065129 997 external_mini_cluster.cc:1428] Setting key 16edb6963a904aaf24494db7505f618f
I20260925 00:52:25.065685 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.057707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:40627 (local address 127.0.249.66:58489)
0925 00:52:25.058353 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:25.058434 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:25.059906 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:25.060427 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:25.060433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:25.060515 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:25.060988 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.061000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.061883 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.061886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.062756 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.062764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.063242 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.063246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.063401 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.063950 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:52:25.063952 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:25.063988 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:52:25.064044 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:25.064349 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:25.064359 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:25.064371 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 00:52:25.064495 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 00:52:25.064777 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260925 00:52:25.066845 21059 heartbeater.cc:347] Connected to a master server at 127.0.249.126:40627
I20260925 00:52:25.067070 21059 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:25.067348 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.249.67:0
--local_ip_for_outbound_sockets=127.0.249.67
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40627
--builtin_ntp_servers=127.0.249.84:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 00:52:25.067623 21059 heartbeater.cc:511] Master 127.0.249.126:40627 requested a full tablet report, sending...
I20260925 00:52:25.068591 20739 ts_manager.cc:194] Registered new tserver with Master: 16bffc3097e847499f2801c8fbdfe71e (127.0.249.66:44859)
I20260925 00:52:25.069147 20739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.66:58489
W20260925 00:52:25.175388 21062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:25.175624 21062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:25.175693 21062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:25.179642 21062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:25.179695 21062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:25.179790 21062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.249.67
I20260925 00:52:25.184566 21062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.249.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.249.67
--webserver_port=0
--tserver_master_addrs=127.0.249.126:40627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.249.67
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:25.185659 21062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:25.187072 21062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:25.196048 21068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:25.196090 21067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:25.196084 21070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:25.196329 21062 server_base.cc:1034] running on GCE node
I20260925 00:52:25.196699 21062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:25.197332 21062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:25.198527 21062 hybrid_clock.cc:648] HybridClock initialized: now 1790297545198512 us; error 27 us; skew 500 ppm
I20260925 00:52:25.203413 21062 webserver.cc:460] Webserver started at http://127.0.249.67:39441/ using document root <none> and password file <none>
I20260925 00:52:25.203986 21062 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:25.204077 21062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:25.204315 21062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:25.206355 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/data/instance:
uuid: "c36592ba035f4d03ac100af61f750fe9"
format_stamp: "Formatted at 2026-09-25 00:52:25 on dist-test-slave-05m0"
server_key: "2144dd68b8b71ee2773cb841651d1b0b"
server_key_iv: "c2cade46748354077708657a9fc08dda"
server_key_version: "encryptionkey@0"
I20260925 00:52:25.206898 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/wal/instance:
uuid: "c36592ba035f4d03ac100af61f750fe9"
format_stamp: "Formatted at 2026-09-25 00:52:25 on dist-test-slave-05m0"
server_key: "2144dd68b8b71ee2773cb841651d1b0b"
server_key_iv: "c2cade46748354077708657a9fc08dda"
server_key_version: "encryptionkey@0"
I20260925 00:52:25.210381 21062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:52:25.212674 21076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:25.213924 21062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:25.214052 21062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/wal
uuid: "c36592ba035f4d03ac100af61f750fe9"
format_stamp: "Formatted at 2026-09-25 00:52:25 on dist-test-slave-05m0"
server_key: "2144dd68b8b71ee2773cb841651d1b0b"
server_key_iv: "c2cade46748354077708657a9fc08dda"
server_key_version: "encryptionkey@0"
I20260925 00:52:25.214160 21062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:25.226718 21062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:25.227373 21062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:25.227577 21062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:25.228180 21062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 00:52:25.229121 21062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 00:52:25.229198 21062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:25.229274 21062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 00:52:25.229332 21062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:25.238714 21062 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.67:44443
I20260925 00:52:25.238736 21189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.67:44443 every 8 connection(s)
I20260925 00:52:25.239698 21062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/data/info.pb
I20260925 00:52:25.243633 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 21062
I20260925 00:52:25.243796 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/wal/instance
I20260925 00:52:25.244154 997 external_mini_cluster.cc:1428] Setting key 0b6ef742929d34c85d16926b4f373121
I20260925 00:52:25.249788 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.241687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.67:44231 (local address 127.0.249.126:40627)
0925 00:52:25.241855 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:25.241859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:25.242773 (+ 914us) server_negotiation.cc:407] Connection header received
0925 00:52:25.244447 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:25.244450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:25.244528 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:25.244665 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:25.245913 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.246568 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.247740 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.247975 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.248090 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.249235 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:25.249239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:25.249241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:25.249254 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:25.249284 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:25.249292 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:25.249375 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:25.249579 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 00:52:25.249643 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 00:52:25.250667 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.242021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:40627 (local address 127.0.249.67:44231)
0925 00:52:25.242635 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:25.242673 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:25.244204 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:25.244907 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:25.244914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:25.245028 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:25.245764 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.245783 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.246691 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.246694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.247605 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.247623 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.248125 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.248130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.248317 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.248966 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 00:52:25.248968 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:25.249010 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 00:52:25.249078 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:25.249376 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:25.249387 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:25.249432 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 00:52:25.249566 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 00:52:25.249863 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260925 00:52:25.251821 21190 heartbeater.cc:347] Connected to a master server at 127.0.249.126:40627
I20260925 00:52:25.252049 21190 heartbeater.cc:464] Registering TS with master...
I20260925 00:52:25.252576 21190 heartbeater.cc:511] Master 127.0.249.126:40627 requested a full tablet report, sending...
I20260925 00:52:25.253603 20739 ts_manager.cc:194] Registered new tserver with Master: c36592ba035f4d03ac100af61f750fe9 (127.0.249.67:44443)
I20260925 00:52:25.254179 20739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.249.67:44231
I20260925 00:52:25.258316 997 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 00:52:25.263926 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.260435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.249.126:40627)
0925 00:52:25.260563 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:25.260566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:25.260729 (+ 163us) server_negotiation.cc:407] Connection header received
0925 00:52:25.260825 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:25.260828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:25.260866 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:25.260931 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:25.261175 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.261724 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.262504 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.262709 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.262780 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.263544 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:25.263547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:25.263548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:25.263557 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:25.263572 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:25.263579 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:25.263633 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:25.263777 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 00:52:25.263823 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:52:25.266577 20739 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 00:52:25.269575 20739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 00:52:25.283526 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.277901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:44443 (local address 127.0.0.1:55328)
0925 00:52:25.278549 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:25.278598 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:25.278738 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:25.279767 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:25.279772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:25.279795 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:25.279891 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.279899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.280915 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.280920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.282227 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.282236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.283144 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.283147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.283222 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.283252 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 00:52:25.283327 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 00:52:25.283411 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":35,"thread_start_us":73,"threads_started":1}
I20260925 00:52:25.283785 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.278006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.0.249.67:44443)
0925 00:52:25.279305 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:25.279319 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:25.279344 (+ 25us) server_negotiation.cc:407] Connection header received
0925 00:52:25.279419 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:25.279424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:25.279607 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:25.279776 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:25.280005 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.280802 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.282346 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.283031 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.283143 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.283331 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:25.283383 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 00:52:25.283477 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1197,"thread_start_us":122,"threads_started":1}
I20260925 00:52:25.284195 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.278598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:44589 (local address 127.0.0.1:43896)
0925 00:52:25.279137 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:25.279150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:25.279231 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:25.279931 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:25.279933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:25.279973 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:25.280065 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.280071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.281427 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.281430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.282570 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.282576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.283617 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.283622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.283887 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.283910 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 00:52:25.283982 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 00:52:25.284052 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":27,"thread_start_us":89,"threads_started":1}
I20260925 00:52:25.284811 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.279008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.0.249.65:44589)
0925 00:52:25.279426 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:25.279437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:25.279455 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:52:25.279519 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:25.279523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:25.279664 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:25.279791 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:25.280212 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.281288 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.282684 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.283478 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.283589 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.283784 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:25.284083 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 00:52:25.284188 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":132,"threads_started":1}
I20260925 00:52:25.285486 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.278015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.66:44859 (local address 127.0.0.1:43960)
0925 00:52:25.278950 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:25.278963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:25.279034 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:25.279558 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:25.279563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:25.279584 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:25.279726 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.279737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.281201 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.281204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.282550 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.282562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.285073 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.285076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.285170 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.285192 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 00:52:25.285260 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 00:52:25.285331 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":41,"thread_start_us":121,"threads_started":1}
I20260925 00:52:25.286110 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.278576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.0.249.66:44859)
0925 00:52:25.279034 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:25.279060 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:25.279105 (+ 45us) server_negotiation.cc:407] Connection header received
0925 00:52:25.279190 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:25.279197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:25.279402 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:25.279557 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:25.279863 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.281073 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.283851 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.284838 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.284954 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.285819 (+ 865us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:25.285871 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 00:52:25.285960 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":115,"threads_started":1}
I20260925 00:52:25.286962 21124 tablet_service.cc:1467] Processing CreateTablet for tablet 267ece6f64e24914ab6d01be3bd3097a (DEFAULT_TABLE table=test-table [id=3ce30bfd99bb4b6287f9bf32fce56ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:25.287985 21124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 267ece6f64e24914ab6d01be3bd3097a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:25.287578 20862 tablet_service.cc:1467] Processing CreateTablet for tablet 267ece6f64e24914ab6d01be3bd3097a (DEFAULT_TABLE table=test-table [id=3ce30bfd99bb4b6287f9bf32fce56ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:25.288621 20862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 267ece6f64e24914ab6d01be3bd3097a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:25.288805 20993 tablet_service.cc:1467] Processing CreateTablet for tablet 267ece6f64e24914ab6d01be3bd3097a (DEFAULT_TABLE table=test-table [id=3ce30bfd99bb4b6287f9bf32fce56ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 00:52:25.289790 20993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 267ece6f64e24914ab6d01be3bd3097a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:25.294220 21207 tablet_bootstrap.cc:492] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: Bootstrap starting.
I20260925 00:52:25.295605 21208 tablet_bootstrap.cc:492] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435: Bootstrap starting.
I20260925 00:52:25.296658 21207 tablet_bootstrap.cc:654] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:25.296887 21209 tablet_bootstrap.cc:492] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: Bootstrap starting.
I20260925 00:52:25.297638 21207 log.cc:824] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:25.297783 21208 tablet_bootstrap.cc:654] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:25.298537 21208 log.cc:824] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:25.298889 21209 tablet_bootstrap.cc:654] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:25.299642 21209 log.cc:824] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:25.299875 21207 tablet_bootstrap.cc:492] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: No bootstrap required, opened a new log
I20260925 00:52:25.300065 21207 ts_tablet_manager.cc:1397] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260925 00:52:25.300421 21208 tablet_bootstrap.cc:492] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435: No bootstrap required, opened a new log
I20260925 00:52:25.300633 21208 ts_tablet_manager.cc:1397] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:52:25.301385 21209 tablet_bootstrap.cc:492] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: No bootstrap required, opened a new log
I20260925 00:52:25.301625 21209 ts_tablet_manager.cc:1397] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:52:25.304560 21209 raft_consensus.cc:348] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } }
I20260925 00:52:25.304505 21207 raft_consensus.cc:348] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } }
I20260925 00:52:25.304790 21207 raft_consensus.cc:374] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:25.304790 21209 raft_consensus.cc:374] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:25.304852 21209 raft_consensus.cc:729] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bffc3097e847499f2801c8fbdfe71e, State: Initialized, Role: FOLLOWER
I20260925 00:52:25.304853 21207 raft_consensus.cc:729] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36592ba035f4d03ac100af61f750fe9, State: Initialized, Role: FOLLOWER
I20260925 00:52:25.304728 21208 raft_consensus.cc:348] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } }
I20260925 00:52:25.304944 21208 raft_consensus.cc:374] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:25.304994 21208 raft_consensus.cc:729] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0feec57afb5a4cf59353dd5a4d79a435, State: Initialized, Role: FOLLOWER
I20260925 00:52:25.305310 21207 consensus_queue.cc:260] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } }
I20260925 00:52:25.305408 21208 consensus_queue.cc:260] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } }
I20260925 00:52:25.305315 21209 consensus_queue.cc:260] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } }
I20260925 00:52:25.306317 21208 ts_tablet_manager.cc:1428] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 00:52:25.306317 21207 ts_tablet_manager.cc:1428] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260925 00:52:25.306725 21190 heartbeater.cc:503] Master 127.0.249.126:40627 was elected leader, sending a full tablet report...
I20260925 00:52:25.306908 21209 ts_tablet_manager.cc:1428] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 00:52:25.307277 21059 heartbeater.cc:503] Master 127.0.249.126:40627 was elected leader, sending a full tablet report...
I20260925 00:52:25.307296 20928 heartbeater.cc:503] Master 127.0.249.126:40627 was elected leader, sending a full tablet report...
W20260925 00:52:25.377324 20929 tablet.cc:2369] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:25.383773 21215 raft_consensus.cc:482] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 00:52:25.383962 21215 raft_consensus.cc:504] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } }
I20260925 00:52:25.385238 21215 leader_election.cc:288] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c36592ba035f4d03ac100af61f750fe9 (127.0.249.67:44443), 0feec57afb5a4cf59353dd5a4d79a435 (127.0.249.65:44589)
I20260925 00:52:25.389341 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.385541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.67:44443 (local address 127.0.249.66:49131)
0925 00:52:25.385764 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:25.385782 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:25.385901 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:25.386221 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:25.386224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:25.386242 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:25.386371 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.386378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.387360 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.387364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.388205 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.388214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.388907 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.388912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.389022 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.389046 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:52:25.389122 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 00:52:25.389189 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 00:52:25.389367 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.385633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:49131 (local address 127.0.249.67:44443)
0925 00:52:25.385769 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:25.385774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:25.385865 (+ 91us) server_negotiation.cc:407] Connection header received
0925 00:52:25.386012 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:25.386017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:25.386092 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:25.386177 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:25.386523 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.387256 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.388337 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.388793 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.388886 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.388925 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:25.389156 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 00:52:25.389243 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:52:25.389499 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.385622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.249.66:40067 (local address 127.0.249.65:44589)
0925 00:52:25.385769 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:25.385774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:25.386118 (+ 344us) server_negotiation.cc:407] Connection header received
0925 00:52:25.386238 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:25.386241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:25.386291 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:25.386376 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 00:52:25.386758 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.387466 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.388496 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.388908 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.389023 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.389065 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:25.389283 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 00:52:25.389351 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:52:25.389516 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.385576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:44589 (local address 127.0.249.66:40067)
0925 00:52:25.386003 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:25.386017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:25.386131 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:25.386413 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:25.386415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:25.386463 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 00:52:25.386639 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.386647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.387574 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.387577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.388376 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:25.388385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.389044 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.389047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:25.389148 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.389172 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 00:52:25.389246 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 00:52:25.389308 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":178,"threads_started":1}
I20260925 00:52:25.390038 21144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ece6f64e24914ab6d01be3bd3097a" candidate_uuid: "16bffc3097e847499f2801c8fbdfe71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c36592ba035f4d03ac100af61f750fe9" is_pre_election: true
I20260925 00:52:25.390118 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ece6f64e24914ab6d01be3bd3097a" candidate_uuid: "16bffc3097e847499f2801c8fbdfe71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0feec57afb5a4cf59353dd5a4d79a435" is_pre_election: true
I20260925 00:52:25.390352 21144 raft_consensus.cc:2454] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16bffc3097e847499f2801c8fbdfe71e in term 0.
I20260925 00:52:25.390403 20882 raft_consensus.cc:2454] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16bffc3097e847499f2801c8fbdfe71e in term 0.
I20260925 00:52:25.390893 20946 leader_election.cc:302] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0feec57afb5a4cf59353dd5a4d79a435, 16bffc3097e847499f2801c8fbdfe71e, c36592ba035f4d03ac100af61f750fe9; no voters:
I20260925 00:52:25.391150 21215 raft_consensus.cc:2781] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 00:52:25.391227 21215 raft_consensus.cc:482] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 00:52:25.391260 21215 raft_consensus.cc:3037] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:25.392329 21215 raft_consensus.cc:504] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } }
I20260925 00:52:25.392693 21215 leader_election.cc:288] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [CANDIDATE]: Term 1 election: Requested vote from peers c36592ba035f4d03ac100af61f750fe9 (127.0.249.67:44443), 0feec57afb5a4cf59353dd5a4d79a435 (127.0.249.65:44589)
I20260925 00:52:25.393009 21144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ece6f64e24914ab6d01be3bd3097a" candidate_uuid: "16bffc3097e847499f2801c8fbdfe71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c36592ba035f4d03ac100af61f750fe9"
I20260925 00:52:25.393132 21144 raft_consensus.cc:3037] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:25.393142 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ece6f64e24914ab6d01be3bd3097a" candidate_uuid: "16bffc3097e847499f2801c8fbdfe71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0feec57afb5a4cf59353dd5a4d79a435"
I20260925 00:52:25.393277 20882 raft_consensus.cc:3037] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:25.394186 21144 raft_consensus.cc:2454] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16bffc3097e847499f2801c8fbdfe71e in term 1.
I20260925 00:52:25.394266 20882 raft_consensus.cc:2454] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16bffc3097e847499f2801c8fbdfe71e in term 1.
I20260925 00:52:25.394479 20946 leader_election.cc:302] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16bffc3097e847499f2801c8fbdfe71e, c36592ba035f4d03ac100af61f750fe9; no voters:
I20260925 00:52:25.394632 21215 raft_consensus.cc:2781] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:25.394842 21215 raft_consensus.cc:686] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 1 LEADER]: Becoming Leader. State: Replica: 16bffc3097e847499f2801c8fbdfe71e, State: Running, Role: LEADER
I20260925 00:52:25.395181 21215 consensus_queue.cc:237] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } }
I20260925 00:52:25.398367 20739 catalog_manager.cc:5495] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 16bffc3097e847499f2801c8fbdfe71e (127.0.249.66). New cstate: current_term: 1 leader_uuid: "16bffc3097e847499f2801c8fbdfe71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16bffc3097e847499f2801c8fbdfe71e" member_type: VOTER last_known_addr { host: "127.0.249.66" port: 44859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 } health_report { overall_health: UNKNOWN } } }
I20260925 00:52:25.434221 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.431048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.249.66:44859)
0925 00:52:25.431174 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:25.431178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:25.431192 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:52:25.431274 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:25.431276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:25.431333 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:25.431423 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:25.431625 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.432114 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.432911 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:25.433113 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:25.433192 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:25.433485 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:25.433901 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:25.433993 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:25.434049 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 00:52:25.434094 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 00:52:25.440188 20882 raft_consensus.cc:1261] T 267ece6f64e24914ab6d01be3bd3097a P 0feec57afb5a4cf59353dd5a4d79a435 [term 1 FOLLOWER]: Refusing update from remote peer 16bffc3097e847499f2801c8fbdfe71e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:25.440188 21144 raft_consensus.cc:1261] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [term 1 FOLLOWER]: Refusing update from remote peer 16bffc3097e847499f2801c8fbdfe71e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:25.440905 21215 consensus_queue.cc:1035] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0feec57afb5a4cf59353dd5a4d79a435" member_type: VOTER last_known_addr { host: "127.0.249.65" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:25.441103 21217 consensus_queue.cc:1035] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36592ba035f4d03ac100af61f750fe9" member_type: VOTER last_known_addr { host: "127.0.249.67" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:25.448889 21222 mvcc.cc:205] Tried to move back new op lower bound from 7333058746114908160 to 7333058745941192704. Current Snapshot: MvccSnapshot[applied={T|T < 7333058746114908160}]
I20260925 00:52:25.449955 21221 mvcc.cc:205] Tried to move back new op lower bound from 7333058746114908160 to 7333058745941192704. Current Snapshot: MvccSnapshot[applied={T|T < 7333058746114908160}]
I20260925 00:52:25.456825 20740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:25.456974 20740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 00:52:25.459525 20740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451b33badfff48c28dfc01172221488d: Sending DeleteTablet for 3 replicas of tablet 267ece6f64e24914ab6d01be3bd3097a
I20260925 00:52:25.460290 21124 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ece6f64e24914ab6d01be3bd3097a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:25 UTC) from {username='slave'} at 127.0.0.1:55328
I20260925 00:52:25.460357 20862 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ece6f64e24914ab6d01be3bd3097a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:25 UTC) from {username='slave'} at 127.0.0.1:43896
I20260925 00:52:25.460325 20993 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ece6f64e24914ab6d01be3bd3097a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 00:52:25 UTC) from {username='slave'} at 127.0.0.1:43960
I20260925 00:52:25.460985 21230 tablet_replica.cc:330] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: stopping tablet replica
I20260925 00:52:25.461316 21230 raft_consensus.cc:2229] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 00:52:25.461588 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 20800
I20260925 00:52:25.461663 21230 raft_consensus.cc:2258] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:25.462498 21231 tablet_replica.cc:330] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: stopping tablet replica
I20260925 00:52:25.462778 21231 raft_consensus.cc:2229] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 1 LEADER]: Raft consensus shutting down.
I20260925 00:52:25.463107 21231 raft_consensus.cc:2258] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 00:52:25.463614 21230 ts_tablet_manager.cc:1905] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:25.464486 21231 ts_tablet_manager.cc:1905] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 00:52:25.465929 21230 ts_tablet_manager.cc:1918] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:25.466035 21230 log.cc:1196] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-2/wal/wals/267ece6f64e24914ab6d01be3bd3097a
I20260925 00:52:25.466312 21230 ts_tablet_manager.cc:1939] T 267ece6f64e24914ab6d01be3bd3097a P c36592ba035f4d03ac100af61f750fe9: Deleting consensus metadata
I20260925 00:52:25.467455 20724 catalog_manager.cc:4841] TS c36592ba035f4d03ac100af61f750fe9 (127.0.249.67:44443): tablet 267ece6f64e24914ab6d01be3bd3097a (table test-table [id=3ce30bfd99bb4b6287f9bf32fce56ddc]) successfully deleted
I20260925 00:52:25.467912 21231 ts_tablet_manager.cc:1918] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 00:52:25.468011 21231 log.cc:1196] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: Deleting WAL directory at /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790297292685301-997-0/minicluster-data/ts-1/wal/wals/267ece6f64e24914ab6d01be3bd3097a
I20260925 00:52:25.468333 21231 ts_tablet_manager.cc:1939] T 267ece6f64e24914ab6d01be3bd3097a P 16bffc3097e847499f2801c8fbdfe71e: Deleting consensus metadata
I20260925 00:52:25.469492 20725 catalog_manager.cc:4841] TS 16bffc3097e847499f2801c8fbdfe71e (127.0.249.66:44859): tablet 267ece6f64e24914ab6d01be3bd3097a (table test-table [id=3ce30bfd99bb4b6287f9bf32fce56ddc]) successfully deleted
W20260925 00:52:25.470327 20725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.249.65:44589 (error 108)
I20260925 00:52:25.470448 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 20931
I20260925 00:52:25.471762 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:25.471380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.65:44589 (local address 127.0.0.1:43902)
0925 00:52:25.471571 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:25.471648 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.65:44589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260925 00:52:25.471923 20725 catalog_manager.cc:4584] TS 0feec57afb5a4cf59353dd5a4d79a435 (127.0.249.65:44589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 267ece6f64e24914ab6d01be3bd3097a: Network error: Client connection negotiation failed: client connection to 127.0.249.65:44589: connect: Connection refused (error 111)
I20260925 00:52:25.475651 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 21062
I20260925 00:52:25.480799 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 20709
2026-09-25T00:52:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1046 ms)
[----------] 6 tests from EncryptionPolicyTest (10210 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:25 dist-test-slave-05m0 krb5kdc[21236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:25 dist-test-slave-05m0 krb5kdc[21236](info): set up 2 sockets
Sep 25 00:52:25 dist-test-slave-05m0 krb5kdc[21236](info): commencing operation
krb5kdc: starting...
W20260925 00:52:26.513646 997 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:26 dist-test-slave-05m0 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:26.633001 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:26.740792 21249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:26.741102 21249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:26.741185 21249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:26.745167 21249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:26.745261 21249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:26.745301 21249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:26.745358 21249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:26.745410 21249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:26.750666 21249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33881
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:26.751727 21249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:26.752765 21249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:26.761035 21257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:26.761109 21255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:26.761407 21249 server_base.cc:1034] running on GCE node
W20260925 00:52:26.761042 21254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:26.762010 21249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:26.763062 21249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:26.764264 21249 hybrid_clock.cc:648] HybridClock initialized: now 1790297546764228 us; error 39 us; skew 500 ppm
Sep 25 00:52:26 dist-test-slave-05m0 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:26.766988 21249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:26.767812 21249 webserver.cc:460] Webserver started at http://127.0.0.1:42819/ using document root <none> and password file <none>
I20260925 00:52:26.768355 21249 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:26.768416 21249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:26.768574 21249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:26.770504 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "29e90f2457d243ba888547e3af6be7ed"
format_stamp: "Formatted at 2026-09-25 00:52:26 on dist-test-slave-05m0"
server_key: "f4affaa721345bc40cba3a6a4291a8a8"
server_key_iv: "c1ff67f59268cad2895fcc6224ada785"
server_key_version: "encryptionkey@0"
I20260925 00:52:26.770999 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "29e90f2457d243ba888547e3af6be7ed"
format_stamp: "Formatted at 2026-09-25 00:52:26 on dist-test-slave-05m0"
server_key: "f4affaa721345bc40cba3a6a4291a8a8"
server_key_iv: "c1ff67f59268cad2895fcc6224ada785"
server_key_version: "encryptionkey@0"
I20260925 00:52:26.774408 21249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:52:26.776729 21264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:26.777978 21249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:26.778112 21249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "29e90f2457d243ba888547e3af6be7ed"
format_stamp: "Formatted at 2026-09-25 00:52:26 on dist-test-slave-05m0"
server_key: "f4affaa721345bc40cba3a6a4291a8a8"
server_key_iv: "c1ff67f59268cad2895fcc6224ada785"
server_key_version: "encryptionkey@0"
I20260925 00:52:26.778215 21249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:26.790122 21249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:26.793151 21249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:26.793296 21249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:26.800452 21249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33881
I20260925 00:52:26.800540 21316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33881 every 8 connection(s)
I20260925 00:52:26.801503 21249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:26.804226 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:26.808674 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 21249
I20260925 00:52:26.808812 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:26.809070 997 external_mini_cluster.cc:1428] Setting key de85d08d0b1e71ee2690104068bb8282
I20260925 00:52:26.809880 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed: Bootstrap starting.
I20260925 00:52:26.812158 21317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:26.813113 21317 log.cc:824] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:26.815160 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed: No bootstrap required, opened a new log
Sep 25 00:52:26 dist-test-slave-05m0 krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 00:52:26.819360 21317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e90f2457d243ba888547e3af6be7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33881 } }
I20260925 00:52:26.819566 21317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:26.819633 21317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e90f2457d243ba888547e3af6be7ed, State: Initialized, Role: FOLLOWER
I20260925 00:52:26.820072 21317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e90f2457d243ba888547e3af6be7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33881 } }
I20260925 00:52:26.820226 21317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:26.820312 21317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:26.820433 21317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:26.821267 21317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e90f2457d243ba888547e3af6be7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33881 } }
I20260925 00:52:26.821626 21317 leader_election.cc:302] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29e90f2457d243ba888547e3af6be7ed; no voters:
I20260925 00:52:26.821955 21317 leader_election.cc:288] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:26.822014 21322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:26.822207 21322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [term 1 LEADER]: Becoming Leader. State: Replica: 29e90f2457d243ba888547e3af6be7ed, State: Running, Role: LEADER
I20260925 00:52:26.822490 21322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e90f2457d243ba888547e3af6be7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33881 } }
I20260925 00:52:26.823204 21317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:26.823593 21322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29e90f2457d243ba888547e3af6be7ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e90f2457d243ba888547e3af6be7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33881 } } }
I20260925 00:52:26.823707 21322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:26.824002 21323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29e90f2457d243ba888547e3af6be7ed. Latest consensus state: current_term: 1 leader_uuid: "29e90f2457d243ba888547e3af6be7ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e90f2457d243ba888547e3af6be7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33881 } } }
I20260925 00:52:26.824093 21323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:26.824450 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:26.810401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58021 (local address 127.0.0.1:33881)
0925 00:52:26.810793 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:26.810810 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:26.810843 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:52:26.811431 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:26.811447 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:26.811779 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:26.812042 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:26.813025 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:26.814309 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:26.815421 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:26.815875 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:26.815989 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:26.818755 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:26.818760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:26.818772 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:26.818797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:26.820716 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:26.821212 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:26.821224 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:26.821229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:26.821333 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:26.821727 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:26.821731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:26.821733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:26.822173 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:26.822992 (+ 819us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:26.823144 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 00:52:26.823370 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":124,"threads_started":1}
I20260925 00:52:26.827517 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 00:52:26.828382 21337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:26.828459 21337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:26.828860 21326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:26.829752 21326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:26.836542 21326 catalog_manager.cc:1347] Generated new cluster ID: d6b66f406e6140beb29cf8d6b95c330e
I20260925 00:52:26.836603 21326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:26.840670 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:26.829938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34185 (local address 127.0.0.1:33881)
0925 00:52:26.830077 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:26.830081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:26.831843 (+ 1762us) server_negotiation.cc:407] Connection header received
0925 00:52:26.831953 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:26.831956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:26.832001 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:26.832084 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:26.832896 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:26.833926 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:26.835070 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:26.835298 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:26.835402 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:26.837116 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:26.837120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:26.837123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:26.837147 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:26.838887 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:26.839387 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:26.839391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:26.839393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:26.839478 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:26.839826 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:26.839829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:26.839831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:26.840031 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:26.840167 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:26.840426 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 00:52:26.840519 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 00:52:26.849287 21326 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:26.850219 21326 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:26.853636 21326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29e90f2457d243ba888547e3af6be7ed: Generated new TSK 0
I20260925 00:52:26.854254 21326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:26.859448 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 21249
W20260925 00:52:26.864840 21234 connection.cc:535] client connection to 127.0.0.1:33881 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33881): Connection reset by peer (error 104)
2026-09-25T00:52:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:26 dist-test-slave-05m0 krb5kdc[21348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:26 dist-test-slave-05m0 krb5kdc[21348](info): set up 2 sockets
Sep 25 00:52:26 dist-test-slave-05m0 krb5kdc[21348](info): commencing operation
krb5kdc: starting...
W20260925 00:52:27.902124 997 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:27 dist-test-slave-05m0 krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:28.024140 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:28.131637 21361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:28.131883 21361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:28.131923 21361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:28.135933 21361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:28.135987 21361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:28.136008 21361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:28.136023 21361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:28.136076 21361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:28.141808 21361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:28.142798 21361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:28.143757 21361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:28.152272 21366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:28.152297 21361 server_base.cc:1034] running on GCE node
W20260925 00:52:28.152209 21367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:28.152205 21369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:28.153051 21361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:28.154260 21361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:28.155468 21361 hybrid_clock.cc:648] HybridClock initialized: now 1790297548155438 us; error 40 us; skew 500 ppm
Sep 25 00:52:28 dist-test-slave-05m0 krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:28.158450 21361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:28.159310 21361 webserver.cc:460] Webserver started at http://127.0.0.1:33643/ using document root <none> and password file <none>
I20260925 00:52:28.159874 21361 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:28.159970 21361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:28.160236 21361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:28.162395 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "b459f48ab1bd40b1bc7146fc8cba0187"
format_stamp: "Formatted at 2026-09-25 00:52:28 on dist-test-slave-05m0"
server_key: "aad4d94a8da4f668dd549f538f82e59e"
server_key_iv: "50df94cd5af24afc546ff23c40006e2f"
server_key_version: "encryptionkey@0"
I20260925 00:52:28.162973 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "b459f48ab1bd40b1bc7146fc8cba0187"
format_stamp: "Formatted at 2026-09-25 00:52:28 on dist-test-slave-05m0"
server_key: "aad4d94a8da4f668dd549f538f82e59e"
server_key_iv: "50df94cd5af24afc546ff23c40006e2f"
server_key_version: "encryptionkey@0"
I20260925 00:52:28.166715 21361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:52:28.169181 21376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:28.170487 21361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:28.170644 21361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "b459f48ab1bd40b1bc7146fc8cba0187"
format_stamp: "Formatted at 2026-09-25 00:52:28 on dist-test-slave-05m0"
server_key: "aad4d94a8da4f668dd549f538f82e59e"
server_key_iv: "50df94cd5af24afc546ff23c40006e2f"
server_key_version: "encryptionkey@0"
I20260925 00:52:28.170751 21361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:28.186309 21361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:28.189919 21361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:28.190132 21361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:28.197444 21361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33315
I20260925 00:52:28.197533 21428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33315 every 8 connection(s)
I20260925 00:52:28.198457 21361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:28.199901 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 21361
I20260925 00:52:28.200079 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:28.200385 997 external_mini_cluster.cc:1428] Setting key 80fef360a78edc42f77eb579a5a8cfb4
I20260925 00:52:28.201486 21429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:28.207947 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187: Bootstrap starting.
I20260925 00:52:28.210481 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:28.201843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60417 (local address 127.0.0.1:33315)
0925 00:52:28.202293 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:28.202311 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:28.202351 (+ 40us) server_negotiation.cc:407] Connection header received
0925 00:52:28.203158 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:28.203191 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:28.203588 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:28.203937 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:28.204346 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:28.205811 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:28.206786 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:28.207464 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:28.207625 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:28.208510 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:28.208516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:28.208529 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:28.208988 (+ 459us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:28.209062 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:28.209092 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:28.209243 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:28.209490 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 00:52:28.209611 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
I20260925 00:52:28.210510 21429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:28.211241 21429 log.cc:824] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:28.213171 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187: No bootstrap required, opened a new log
I20260925 00:52:28.213647 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:28.217820 21429 raft_consensus.cc:348] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459f48ab1bd40b1bc7146fc8cba0187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } }
I20260925 00:52:28.218091 21429 raft_consensus.cc:374] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:28.218148 21429 raft_consensus.cc:729] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b459f48ab1bd40b1bc7146fc8cba0187, State: Initialized, Role: FOLLOWER
I20260925 00:52:28.218602 21429 consensus_queue.cc:260] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459f48ab1bd40b1bc7146fc8cba0187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } }
I20260925 00:52:28.218789 21429 raft_consensus.cc:388] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:28.218864 21429 raft_consensus.cc:482] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:28.218982 21429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:28.220009 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:28.215915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49219 (local address 127.0.0.1:33315)
0925 00:52:28.216053 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:28.216058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:28.216268 (+ 210us) server_negotiation.cc:407] Connection header received
0925 00:52:28.216358 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:28.216360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:28.216398 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:28.216463 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:28.216756 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:28.217236 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:28.218350 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:28.218586 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:28.218668 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:28.219508 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:28.219511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:28.219513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:28.219524 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:28.219547 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:28.219557 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:28.219668 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:28.219814 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 00:52:28.219845 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 00:52:28.220242 21429 raft_consensus.cc:504] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459f48ab1bd40b1bc7146fc8cba0187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } }
I20260925 00:52:28.220602 21429 leader_election.cc:302] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b459f48ab1bd40b1bc7146fc8cba0187; no voters:
I20260925 00:52:28.220935 21429 leader_election.cc:288] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:28.221000 21439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:28.221266 21439 raft_consensus.cc:686] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [term 1 LEADER]: Becoming Leader. State: Replica: b459f48ab1bd40b1bc7146fc8cba0187, State: Running, Role: LEADER
I20260925 00:52:28.221611 21439 consensus_queue.cc:237] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459f48ab1bd40b1bc7146fc8cba0187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } }
I20260925 00:52:28.222303 21429 sys_catalog.cc:564] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:28.223457 21441 sys_catalog.cc:455] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b459f48ab1bd40b1bc7146fc8cba0187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459f48ab1bd40b1bc7146fc8cba0187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } } }
I20260925 00:52:28.223472 21440 sys_catalog.cc:455] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b459f48ab1bd40b1bc7146fc8cba0187. Latest consensus state: current_term: 1 leader_uuid: "b459f48ab1bd40b1bc7146fc8cba0187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b459f48ab1bd40b1bc7146fc8cba0187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } } }
I20260925 00:52:28.223685 21441 sys_catalog.cc:458] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:28.223721 21440 sys_catalog.cc:458] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:28.224280 21450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:28.226977 21450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:28.232736 21450 catalog_manager.cc:1347] Generated new cluster ID: 1f7973365d904e1280d4d47700308cf9
I20260925 00:52:28.232811 21450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:28.247177 21450 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:28.248478 21450 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:28.253152 21450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b459f48ab1bd40b1bc7146fc8cba0187: Generated new TSK 0
I20260925 00:52:28.253984 21450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:28.257563 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 21361
W20260925 00:52:28.262883 21346 connection.cc:535] client connection to 127.0.0.1:33315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33315): Connection reset by peer (error 104)
2026-09-25T00:52:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:28 dist-test-slave-05m0 krb5kdc[21460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:28 dist-test-slave-05m0 krb5kdc[21460](info): set up 2 sockets
Sep 25 00:52:28 dist-test-slave-05m0 krb5kdc[21460](info): commencing operation
krb5kdc: starting...
W20260925 00:52:29.297818 997 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:29 dist-test-slave-05m0 krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:29.424474 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:29.532881 21473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:29.533129 21473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:29.533169 21473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:29.537017 21473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:29.537070 21473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:29.537088 21473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:29.537109 21473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:29.537124 21473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:29.542171 21473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40399
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:29.543087 21473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:29.543990 21473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:29.552104 21481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:29.552282 21473 server_base.cc:1034] running on GCE node
W20260925 00:52:29.552193 21478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:29.552197 21479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:29.552966 21473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:29.554122 21473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:29.555302 21473 hybrid_clock.cc:648] HybridClock initialized: now 1790297549555271 us; error 43 us; skew 500 ppm
Sep 25 00:52:29 dist-test-slave-05m0 krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:29.558061 21473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:29.558930 21473 webserver.cc:460] Webserver started at http://127.0.0.1:36077/ using document root <none> and password file <none>
I20260925 00:52:29.559469 21473 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:29.559530 21473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:29.559767 21473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:29.561798 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "9b58478279f748419dd5c2539a32cd06"
format_stamp: "Formatted at 2026-09-25 00:52:29 on dist-test-slave-05m0"
server_key: "0d9da900af429b6473cf8f758d47109f"
server_key_iv: "6f5d2456e49b6ae40b5b87d413c1dd44"
server_key_version: "encryptionkey@0"
I20260925 00:52:29.562355 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "9b58478279f748419dd5c2539a32cd06"
format_stamp: "Formatted at 2026-09-25 00:52:29 on dist-test-slave-05m0"
server_key: "0d9da900af429b6473cf8f758d47109f"
server_key_iv: "6f5d2456e49b6ae40b5b87d413c1dd44"
server_key_version: "encryptionkey@0"
I20260925 00:52:29.566046 21473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:52:29.568370 21488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:29.569598 21473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:29.569727 21473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "9b58478279f748419dd5c2539a32cd06"
format_stamp: "Formatted at 2026-09-25 00:52:29 on dist-test-slave-05m0"
server_key: "0d9da900af429b6473cf8f758d47109f"
server_key_iv: "6f5d2456e49b6ae40b5b87d413c1dd44"
server_key_version: "encryptionkey@0"
I20260925 00:52:29.569836 21473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:29.574276 21473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:29.577023 21473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:29.577214 21473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:29.584316 21473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40399
I20260925 00:52:29.584321 21540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40399 every 8 connection(s)
I20260925 00:52:29.585228 21473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:29.587842 21541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:29.590027 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 21473
I20260925 00:52:29.590224 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:29.590499 997 external_mini_cluster.cc:1428] Setting key 27b7832a8568b14e59e5a55fa76d3ab5
I20260925 00:52:29.593729 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06: Bootstrap starting.
I20260925 00:52:29.595975 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:29.591905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54979 (local address 127.0.0.1:40399)
0925 00:52:29.592366 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:29.592380 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:29.592416 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:52:29.593071 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:29.593088 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:29.593362 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:29.593840 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:29.594619 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:29.594623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:29.594634 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:29.594644 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:29.594695 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:29.594719 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:29.594886 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:29.595097 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 00:52:29.595262 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":132,"threads_started":1}
I20260925 00:52:29.596285 21541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:29.597012 21541 log.cc:824] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:29.598765 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:29.598984 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06: No bootstrap required, opened a new log
I20260925 00:52:29.602916 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:29.601091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42755 (local address 127.0.0.1:40399)
0925 00:52:29.601228 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:29.601233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:29.601456 (+ 223us) server_negotiation.cc:407] Connection header received
0925 00:52:29.601592 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:29.601595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:29.601626 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:29.601719 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:29.602473 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:29.602477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:29.602479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:29.602490 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:29.602514 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:29.602532 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:29.602615 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:29.602765 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 00:52:29.602800 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 00:52:29.603298 21541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58478279f748419dd5c2539a32cd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40399 } }
I20260925 00:52:29.603531 21541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:29.603606 21541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b58478279f748419dd5c2539a32cd06, State: Initialized, Role: FOLLOWER
I20260925 00:52:29.604074 21541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58478279f748419dd5c2539a32cd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40399 } }
I20260925 00:52:29.604228 21541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:29.604327 21541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:29.604442 21541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:29.605302 21541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58478279f748419dd5c2539a32cd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40399 } }
I20260925 00:52:29.605647 21541 leader_election.cc:302] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b58478279f748419dd5c2539a32cd06; no voters:
I20260925 00:52:29.605932 21541 leader_election.cc:288] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:29.606063 21551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:29.606289 21551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [term 1 LEADER]: Becoming Leader. State: Replica: 9b58478279f748419dd5c2539a32cd06, State: Running, Role: LEADER
I20260925 00:52:29.606608 21551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58478279f748419dd5c2539a32cd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40399 } }
I20260925 00:52:29.607125 21541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:29.608214 21552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b58478279f748419dd5c2539a32cd06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58478279f748419dd5c2539a32cd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40399 } } }
I20260925 00:52:29.608337 21552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:29.608656 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b58478279f748419dd5c2539a32cd06. Latest consensus state: current_term: 1 leader_uuid: "9b58478279f748419dd5c2539a32cd06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58478279f748419dd5c2539a32cd06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40399 } } }
I20260925 00:52:29.608727 21560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:29.608755 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:29.611579 21560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:29.616901 21560 catalog_manager.cc:1347] Generated new cluster ID: 519ca47606924ee686346d3ce998ff24
I20260925 00:52:29.616962 21560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:29.636440 21560 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:29.636790 21560 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:29.647541 21560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b58478279f748419dd5c2539a32cd06: Generated new TSK 0
I20260925 00:52:29.648347 21560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:29.657157 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 21473
2026-09-25T00:52:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:29 dist-test-slave-05m0 krb5kdc[21572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:29 dist-test-slave-05m0 krb5kdc[21572](info): set up 2 sockets
Sep 25 00:52:29 dist-test-slave-05m0 krb5kdc[21572](info): commencing operation
krb5kdc: starting...
W20260925 00:52:30.729761 997 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:30 dist-test-slave-05m0 krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:30.854306 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:30.961231 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:30.961514 21585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:30.961556 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:30.965552 21585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:30.965612 21585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:30.965632 21585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:30.965648 21585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:30.965708 21585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:30.970803 21585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:30.971761 21585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:30.972721 21585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:30.980890 21591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:30.980892 21590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:30.980934 21593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:30.981091 21585 server_base.cc:1034] running on GCE node
I20260925 00:52:30.981853 21585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:30.982965 21585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:30.984163 21585 hybrid_clock.cc:648] HybridClock initialized: now 1790297550984145 us; error 35 us; skew 500 ppm
Sep 25 00:52:30 dist-test-slave-05m0 krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:30.990329 21585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:30.991165 21585 webserver.cc:460] Webserver started at http://127.0.0.1:39017/ using document root <none> and password file <none>
I20260925 00:52:30.991660 21585 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:30.991719 21585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:30.991875 21585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:30.993849 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "a07066b6ad154eaabc17c9293a64435f"
format_stamp: "Formatted at 2026-09-25 00:52:30 on dist-test-slave-05m0"
server_key: "60252ae738dd34248c6bb48a753e4023"
server_key_iv: "499d42e5cb689d93efa34c7d0f4a6131"
server_key_version: "encryptionkey@0"
I20260925 00:52:30.994345 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "a07066b6ad154eaabc17c9293a64435f"
format_stamp: "Formatted at 2026-09-25 00:52:30 on dist-test-slave-05m0"
server_key: "60252ae738dd34248c6bb48a753e4023"
server_key_iv: "499d42e5cb689d93efa34c7d0f4a6131"
server_key_version: "encryptionkey@0"
I20260925 00:52:30.997817 21585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:52:31.000159 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:31.001361 21585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:31.001554 21585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "a07066b6ad154eaabc17c9293a64435f"
format_stamp: "Formatted at 2026-09-25 00:52:30 on dist-test-slave-05m0"
server_key: "60252ae738dd34248c6bb48a753e4023"
server_key_iv: "499d42e5cb689d93efa34c7d0f4a6131"
server_key_version: "encryptionkey@0"
I20260925 00:52:31.001657 21585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:31.023108 21585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:31.025959 21585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:31.026160 21585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:31.033090 21585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43553
I20260925 00:52:31.033113 21652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43553 every 8 connection(s)
I20260925 00:52:31.034070 21585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:31.036669 21653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:31.039896 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 21585
I20260925 00:52:31.040040 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:31.040349 997 external_mini_cluster.cc:1428] Setting key 4a0f00cd12f71e0ea6419ea05f146a09
I20260925 00:52:31.042451 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f: Bootstrap starting.
I20260925 00:52:31.044566 21653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:31.045280 21653 log.cc:824] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:31.047016 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f: No bootstrap required, opened a new log
Sep 25 00:52:31 dist-test-slave-05m0 krb5kdc[21572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 00:52:31.050889 21653 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07066b6ad154eaabc17c9293a64435f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } }
I20260925 00:52:31.051121 21653 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:31.051211 21653 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07066b6ad154eaabc17c9293a64435f, State: Initialized, Role: FOLLOWER
I20260925 00:52:31.051677 21653 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07066b6ad154eaabc17c9293a64435f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } }
I20260925 00:52:31.051832 21653 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:31.051905 21653 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:31.052011 21653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:31.053027 21653 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07066b6ad154eaabc17c9293a64435f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } }
I20260925 00:52:31.053357 21653 leader_election.cc:302] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07066b6ad154eaabc17c9293a64435f; no voters:
I20260925 00:52:31.053695 21653 leader_election.cc:288] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:31.053797 21658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:31.054015 21658 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [term 1 LEADER]: Becoming Leader. State: Replica: a07066b6ad154eaabc17c9293a64435f, State: Running, Role: LEADER
I20260925 00:52:31.054319 21658 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07066b6ad154eaabc17c9293a64435f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } }
I20260925 00:52:31.054481 21656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:31.041773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57205 (local address 127.0.0.1:43553)
0925 00:52:31.042191 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:31.042208 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:31.042242 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:52:31.042908 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:31.042925 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:31.043255 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:31.043574 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:31.044404 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:31.045495 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:31.046343 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:31.046813 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:31.047000 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:31.049474 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:31.049479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:31.049492 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:31.049517 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:31.051189 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:31.051632 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:31.051638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:31.051652 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:31.051780 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:31.052140 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:31.052144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:31.052146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:31.052523 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:31.052673 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:31.052981 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 00:52:31.053384 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
I20260925 00:52:31.054958 21653 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:31.055480 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07066b6ad154eaabc17c9293a64435f. Latest consensus state: current_term: 1 leader_uuid: "a07066b6ad154eaabc17c9293a64435f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07066b6ad154eaabc17c9293a64435f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } } }
I20260925 00:52:31.055586 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:31.055883 21659 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07066b6ad154eaabc17c9293a64435f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07066b6ad154eaabc17c9293a64435f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } } }
I20260925 00:52:31.056002 21659 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:31.056607 21662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:31.057653 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:31.059789 21662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:31.065090 21662 catalog_manager.cc:1347] Generated new cluster ID: 988a38bb280e4c4b9f226bddd9fb6efa
I20260925 00:52:31.065165 21662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:31.069945 21656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:31.059774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59337 (local address 127.0.0.1:43553)
0925 00:52:31.059889 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:31.059893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:31.061898 (+ 2005us) server_negotiation.cc:407] Connection header received
0925 00:52:31.062090 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:31.062093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:31.062138 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:31.062223 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:31.063033 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:31.063719 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:31.064975 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:31.065309 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:31.065452 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:31.066805 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:31.066808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:31.066811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:31.066834 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:31.068303 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:31.068728 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:31.068733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:31.068735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:31.068814 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:31.069098 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:31.069101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:31.069103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:31.069277 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:31.069443 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:31.069660 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 00:52:31.069770 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 00:52:31.080829 21662 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:31.081768 21662 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:31.085776 21662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07066b6ad154eaabc17c9293a64435f: Generated new TSK 0
I20260925 00:52:31.086560 21662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:31.089682 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 21585
2026-09-25T00:52:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:31 dist-test-slave-05m0 krb5kdc[21684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:31 dist-test-slave-05m0 krb5kdc[21684](info): set up 2 sockets
Sep 25 00:52:31 dist-test-slave-05m0 krb5kdc[21684](info): commencing operation
krb5kdc: starting...
W20260925 00:52:32.209678 997 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:32 dist-test-slave-05m0 krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:32.328667 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:32.440884 21697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:32.441195 21697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:32.441269 21697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:32.445426 21697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:32.445514 21697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:32.445552 21697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:32.445608 21697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:32.445645 21697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:32.450975 21697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:32.452068 21697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:32.453135 21697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:32.461675 21702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:32.461683 21703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:32.461737 21697 server_base.cc:1034] running on GCE node
W20260925 00:52:32.461769 21705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:32.462453 21697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:32.463533 21697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:32.464735 21697 hybrid_clock.cc:648] HybridClock initialized: now 1790297552464715 us; error 31 us; skew 500 ppm
Sep 25 00:52:32 dist-test-slave-05m0 krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:32.467468 21697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:32.468402 21697 webserver.cc:460] Webserver started at http://127.0.0.1:43455/ using document root <none> and password file <none>
I20260925 00:52:32.468948 21697 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:32.469038 21697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:32.469274 21697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:32.471516 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "a554e831e3934484843d04608f63422c"
format_stamp: "Formatted at 2026-09-25 00:52:32 on dist-test-slave-05m0"
server_key: "93ed383ea29de6e61c51ede5de83c138"
server_key_iv: "41cef731994f4327d7a62c3fdd3038bc"
server_key_version: "encryptionkey@0"
I20260925 00:52:32.472344 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "a554e831e3934484843d04608f63422c"
format_stamp: "Formatted at 2026-09-25 00:52:32 on dist-test-slave-05m0"
server_key: "93ed383ea29de6e61c51ede5de83c138"
server_key_iv: "41cef731994f4327d7a62c3fdd3038bc"
server_key_version: "encryptionkey@0"
I20260925 00:52:32.476169 21697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 00:52:32.478545 21712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:32.479785 21697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:52:32.479882 21697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "a554e831e3934484843d04608f63422c"
format_stamp: "Formatted at 2026-09-25 00:52:32 on dist-test-slave-05m0"
server_key: "93ed383ea29de6e61c51ede5de83c138"
server_key_iv: "41cef731994f4327d7a62c3fdd3038bc"
server_key_version: "encryptionkey@0"
I20260925 00:52:32.479969 21697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:32.505082 21697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:32.508497 21697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:32.508659 21697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:32.515298 21697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43677
I20260925 00:52:32.515389 21764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43677 every 8 connection(s)
I20260925 00:52:32.516268 21697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:32.518981 21765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:32.524252 21765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c: Bootstrap starting.
I20260925 00:52:32.524642 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 21697
I20260925 00:52:32.524791 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:32.525059 997 external_mini_cluster.cc:1428] Setting key b9c7121488b7cccc367bc7cff4a9eb12
I20260925 00:52:32.526901 21765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:32.527613 21765 log.cc:824] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:32.529548 21765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c: No bootstrap required, opened a new log
I20260925 00:52:32.533826 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:32.526352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58757 (local address 127.0.0.1:43677)
0925 00:52:32.526731 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:32.526748 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:32.526781 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:52:32.527428 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:32.527446 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:32.527839 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:32.528177 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:32.528548 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:32.529548 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:32.530749 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:32.531197 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:32.531310 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:32.532131 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:32.532136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:32.532148 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:32.532508 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:32.532556 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:32.532577 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:32.532652 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:32.532808 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 00:52:32.533026 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":116,"threads_started":1}
I20260925 00:52:32.533700 21765 raft_consensus.cc:348] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554e831e3934484843d04608f63422c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43677 } }
I20260925 00:52:32.533979 21765 raft_consensus.cc:374] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:32.534065 21765 raft_consensus.cc:729] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a554e831e3934484843d04608f63422c, State: Initialized, Role: FOLLOWER
I20260925 00:52:32.534549 21765 consensus_queue.cc:260] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554e831e3934484843d04608f63422c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43677 } }
I20260925 00:52:32.534698 21765 raft_consensus.cc:388] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:32.534782 21765 raft_consensus.cc:482] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:32.534883 21765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:32.535741 21765 raft_consensus.cc:504] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554e831e3934484843d04608f63422c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43677 } }
I20260925 00:52:32.536087 21765 leader_election.cc:302] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a554e831e3934484843d04608f63422c; no voters:
I20260925 00:52:32.536430 21765 leader_election.cc:288] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:32.536491 21770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:32.536549 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:32.536702 21770 raft_consensus.cc:686] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [term 1 LEADER]: Becoming Leader. State: Replica: a554e831e3934484843d04608f63422c, State: Running, Role: LEADER
I20260925 00:52:32.537034 21770 consensus_queue.cc:237] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554e831e3934484843d04608f63422c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43677 } }
I20260925 00:52:32.537699 21765 sys_catalog.cc:564] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:32.538733 21772 sys_catalog.cc:455] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a554e831e3934484843d04608f63422c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554e831e3934484843d04608f63422c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43677 } } }
I20260925 00:52:32.538883 21772 sys_catalog.cc:458] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:32.538940 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a554e831e3934484843d04608f63422c. Latest consensus state: current_term: 1 leader_uuid: "a554e831e3934484843d04608f63422c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554e831e3934484843d04608f63422c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43677 } } }
I20260925 00:52:32.539031 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:32.539305 21784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:32.543504 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:32.538702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55647 (local address 127.0.0.1:43677)
0925 00:52:32.538807 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:32.538811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:32.539128 (+ 317us) server_negotiation.cc:407] Connection header received
0925 00:52:32.539380 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:32.539385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:32.539431 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:32.539480 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:32.539841 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:32.540582 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:32.541817 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:32.542130 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:32.542240 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:32.543076 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:32.543080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:32.543083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:32.543095 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:32.543124 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:32.543137 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:32.543214 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:32.543324 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 00:52:32.543373 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":42}
I20260925 00:52:32.543803 21784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:32.549150 21784 catalog_manager.cc:1347] Generated new cluster ID: 9dc2bee45e1b43aca35c6998c75ef7f5
I20260925 00:52:32.549214 21784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:32.556507 21784 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:32.557529 21784 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:32.565090 21784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a554e831e3934484843d04608f63422c: Generated new TSK 0
I20260925 00:52:32.565896 21784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:32.573690 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 21697
2026-09-25T00:52:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:32 dist-test-slave-05m0 krb5kdc[21796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:32 dist-test-slave-05m0 krb5kdc[21796](info): set up 2 sockets
Sep 25 00:52:32 dist-test-slave-05m0 krb5kdc[21796](info): commencing operation
krb5kdc: starting...
W20260925 00:52:33.617866 997 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:33 dist-test-slave-05m0 krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:33.741256 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:33.850035 21809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:33.850318 21809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:33.850415 21809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:33.854431 21809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:33.854509 21809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:33.854547 21809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:33.854602 21809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:33.854648 21809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:33.859838 21809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:33.860843 21809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:33.861865 21809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:33.870003 21817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:33.870009 21814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:33.869987 21815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:33.870453 21809 server_base.cc:1034] running on GCE node
I20260925 00:52:33.870960 21809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:33.871958 21809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:33.873118 21809 hybrid_clock.cc:648] HybridClock initialized: now 1790297553873094 us; error 40 us; skew 500 ppm
Sep 25 00:52:33 dist-test-slave-05m0 krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:33.875910 21809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:33.876694 21809 webserver.cc:460] Webserver started at http://127.0.0.1:36603/ using document root <none> and password file <none>
I20260925 00:52:33.877199 21809 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:33.877256 21809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:33.877473 21809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:33.879369 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "576da2826b104fea8e90446094e6d979"
format_stamp: "Formatted at 2026-09-25 00:52:33 on dist-test-slave-05m0"
server_key: "e4315d276ea3e32c0e4e0fb5da629932"
server_key_iv: "6079d056fd2873fb633917e53d559ed4"
server_key_version: "encryptionkey@0"
I20260925 00:52:33.879886 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "576da2826b104fea8e90446094e6d979"
format_stamp: "Formatted at 2026-09-25 00:52:33 on dist-test-slave-05m0"
server_key: "e4315d276ea3e32c0e4e0fb5da629932"
server_key_iv: "6079d056fd2873fb633917e53d559ed4"
server_key_version: "encryptionkey@0"
I20260925 00:52:33.883390 21809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 00:52:33.885646 21824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:33.886943 21809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 00:52:33.887104 21809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "576da2826b104fea8e90446094e6d979"
format_stamp: "Formatted at 2026-09-25 00:52:33 on dist-test-slave-05m0"
server_key: "e4315d276ea3e32c0e4e0fb5da629932"
server_key_iv: "6079d056fd2873fb633917e53d559ed4"
server_key_version: "encryptionkey@0"
I20260925 00:52:33.887225 21809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:33.891568 21809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:33.894522 21809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:33.894713 21809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:33.900990 21809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44947
I20260925 00:52:33.901070 21876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44947 every 8 connection(s)
I20260925 00:52:33.901926 21809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:33.904903 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:33.906836 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 21809
I20260925 00:52:33.906996 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:33.907279 997 external_mini_cluster.cc:1428] Setting key ce1b770d4489c9062464259ff048b318
I20260925 00:52:33.912043 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979: Bootstrap starting.
I20260925 00:52:33.912962 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:33.908689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47849 (local address 127.0.0.1:44947)
0925 00:52:33.909154 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:33.909173 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:33.909215 (+ 42us) server_negotiation.cc:407] Connection header received
0925 00:52:33.910114 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:33.910138 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:33.910500 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:33.910885 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:33.911634 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:33.911640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:33.911656 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:33.911667 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:33.911730 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:33.911756 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:33.911895 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:33.911963 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 00:52:33.912127 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":110,"threads_started":1}
I20260925 00:52:33.914681 21877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:33.915551 21877 log.cc:824] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:33.915896 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:33.917511 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979: No bootstrap required, opened a new log
I20260925 00:52:33.919623 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:33.917943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36401 (local address 127.0.0.1:44947)
0925 00:52:33.918067 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:33.918071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:33.918281 (+ 210us) server_negotiation.cc:407] Connection header received
0925 00:52:33.918390 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:33.918392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:33.918419 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:33.918482 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:33.919187 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:33.919190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:33.919192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:33.919203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:33.919230 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:33.919241 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:33.919328 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:33.919426 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 00:52:33.919456 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 00:52:33.921851 21877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576da2826b104fea8e90446094e6d979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260925 00:52:33.922129 21877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:33.922194 21877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576da2826b104fea8e90446094e6d979, State: Initialized, Role: FOLLOWER
I20260925 00:52:33.922633 21877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576da2826b104fea8e90446094e6d979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260925 00:52:33.922755 21877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:33.922839 21877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:33.922953 21877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:33.923818 21877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576da2826b104fea8e90446094e6d979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260925 00:52:33.924157 21877 leader_election.cc:302] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 576da2826b104fea8e90446094e6d979; no voters:
I20260925 00:52:33.924505 21877 leader_election.cc:288] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:33.924572 21887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:33.924839 21887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [term 1 LEADER]: Becoming Leader. State: Replica: 576da2826b104fea8e90446094e6d979, State: Running, Role: LEADER
I20260925 00:52:33.925218 21887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576da2826b104fea8e90446094e6d979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260925 00:52:33.925784 21877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:33.926956 21888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "576da2826b104fea8e90446094e6d979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576da2826b104fea8e90446094e6d979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } } }
I20260925 00:52:33.927104 21888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:33.927371 21889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 576da2826b104fea8e90446094e6d979. Latest consensus state: current_term: 1 leader_uuid: "576da2826b104fea8e90446094e6d979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "576da2826b104fea8e90446094e6d979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } } }
I20260925 00:52:33.927583 21889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:33.927500 21896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:33.930509 21896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:33.935938 21896 catalog_manager.cc:1347] Generated new cluster ID: 6db9e0a592ee462fb6f83877e7ce725b
I20260925 00:52:33.936055 21896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:33.951656 21896 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:33.951995 21896 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:33.962313 21896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 576da2826b104fea8e90446094e6d979: Generated new TSK 0
I20260925 00:52:33.963027 21896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:33.973038 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 21809
2026-09-25T00:52:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:34 dist-test-slave-05m0 krb5kdc[21908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:34 dist-test-slave-05m0 krb5kdc[21908](info): set up 2 sockets
Sep 25 00:52:34 dist-test-slave-05m0 krb5kdc[21908](info): commencing operation
krb5kdc: starting...
W20260925 00:52:35.029642 997 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:35 dist-test-slave-05m0 krb5kdc[21908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:35.150647 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:35.259338 21921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:35.259578 21921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:35.259619 21921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:35.263530 21921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:35.263581 21921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:35.263599 21921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:35.263615 21921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:35.263630 21921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:35.268738 21921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:35.269730 21921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:35.270673 21921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:35.279279 21927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:35.279294 21929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:35.279433 21921 server_base.cc:1034] running on GCE node
W20260925 00:52:35.279294 21926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:35.280016 21921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:35.281100 21921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:35.282280 21921 hybrid_clock.cc:648] HybridClock initialized: now 1790297555282247 us; error 50 us; skew 500 ppm
Sep 25 00:52:35 dist-test-slave-05m0 krb5kdc[21908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:35.284945 21921 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:35.285940 21921 webserver.cc:460] Webserver started at http://127.0.0.1:36475/ using document root <none> and password file <none>
I20260925 00:52:35.286479 21921 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:35.286571 21921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:35.286779 21921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:35.288702 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "18366fabf81b4a74b11d0a73b519e385"
format_stamp: "Formatted at 2026-09-25 00:52:35 on dist-test-slave-05m0"
server_key: "cb1c7e7e1a8dd7c300f3c966de9d1b1d"
server_key_iv: "14c08073a554a459cf356eee5b124e63"
server_key_version: "encryptionkey@0"
I20260925 00:52:35.289249 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "18366fabf81b4a74b11d0a73b519e385"
format_stamp: "Formatted at 2026-09-25 00:52:35 on dist-test-slave-05m0"
server_key: "cb1c7e7e1a8dd7c300f3c966de9d1b1d"
server_key_iv: "14c08073a554a459cf356eee5b124e63"
server_key_version: "encryptionkey@0"
I20260925 00:52:35.292910 21921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:52:35.295243 21936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:35.296471 21921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:35.296612 21921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "18366fabf81b4a74b11d0a73b519e385"
format_stamp: "Formatted at 2026-09-25 00:52:35 on dist-test-slave-05m0"
server_key: "cb1c7e7e1a8dd7c300f3c966de9d1b1d"
server_key_iv: "14c08073a554a459cf356eee5b124e63"
server_key_version: "encryptionkey@0"
I20260925 00:52:35.296741 21921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:35.313690 21921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:35.316520 21921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:35.316721 21921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:35.325423 21921 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35671
I20260925 00:52:35.325433 21988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35671 every 8 connection(s)
I20260925 00:52:35.326610 21921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:35.330185 21989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:35.335661 21989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385: Bootstrap starting.
I20260925 00:52:35.336485 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 21921
I20260925 00:52:35.336633 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:35.336966 997 external_mini_cluster.cc:1428] Setting key e136545430a7fde92ad9e34cf4b73137
I20260925 00:52:35.338631 21989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:35.339401 21989 log.cc:824] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:35.341500 21989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385: No bootstrap required, opened a new log
Sep 25 00:52:35 dist-test-slave-05m0 krb5kdc[21908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 00:52:35.346100 21989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18366fabf81b4a74b11d0a73b519e385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260925 00:52:35.346318 21989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:35.346376 21989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18366fabf81b4a74b11d0a73b519e385, State: Initialized, Role: FOLLOWER
I20260925 00:52:35.346840 21989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18366fabf81b4a74b11d0a73b519e385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260925 00:52:35.347029 21989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:35.347141 21989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:35.347302 21989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:35.348302 21989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18366fabf81b4a74b11d0a73b519e385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260925 00:52:35.348640 21989 leader_election.cc:302] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18366fabf81b4a74b11d0a73b519e385; no voters:
I20260925 00:52:35.348968 21989 leader_election.cc:288] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:35.349036 21994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:35.349265 21994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [term 1 LEADER]: Becoming Leader. State: Replica: 18366fabf81b4a74b11d0a73b519e385, State: Running, Role: LEADER
I20260925 00:52:35.349637 21994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18366fabf81b4a74b11d0a73b519e385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260925 00:52:35.350353 21989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:35.351382 21995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18366fabf81b4a74b11d0a73b519e385. Latest consensus state: current_term: 1 leader_uuid: "18366fabf81b4a74b11d0a73b519e385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18366fabf81b4a74b11d0a73b519e385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } } }
I20260925 00:52:35.351536 21995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:35.351920 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:35.338114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:46097 (local address 127.0.0.1:35671)
0925 00:52:35.338525 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:35.338539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:35.338572 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:52:35.339220 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:35.339236 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:35.339610 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:35.339963 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:35.340799 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:35.341876 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:35.343061 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:35.343501 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:35.343636 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:35.346539 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:35.346544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:35.346557 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:35.346583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:35.348418 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:35.349005 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:35.349011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:35.349017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:35.349203 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:35.349665 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:35.349667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:35.349669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:35.350100 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:35.350329 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:35.350643 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 00:52:35.350891 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260925 00:52:35.351933 22003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:35.353055 21996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18366fabf81b4a74b11d0a73b519e385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18366fabf81b4a74b11d0a73b519e385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } } }
I20260925 00:52:35.353219 21996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:35.355024 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:35.355468 22003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:35.361164 22003 catalog_manager.cc:1347] Generated new cluster ID: da18bbc5cfc443c4ac64a82f0e597244
I20260925 00:52:35.361275 22003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:35.365978 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:35.357205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:52661 (local address 127.0.0.1:35671)
0925 00:52:35.357317 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:35.357323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:35.357490 (+ 167us) server_negotiation.cc:407] Connection header received
0925 00:52:35.357631 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:35.357634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:35.357687 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:35.357770 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:35.358413 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:35.358922 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:35.359738 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:35.360064 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:35.360167 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:35.361965 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:35.361969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:35.361971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:35.361995 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:35.364127 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:35.364692 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:35.364695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:35.364697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:35.364772 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:35.365112 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:35.365114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:35.365116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:35.365305 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:35.365446 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:35.365745 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 00:52:35.365830 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 00:52:35.369859 22003 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:35.370666 22003 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:35.379014 22003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18366fabf81b4a74b11d0a73b519e385: Generated new TSK 0
I20260925 00:52:35.379915 22003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:35.384114 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 21921
W20260925 00:52:35.389441 21906 connection.cc:535] client connection to 127.0.0.1:35671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35671): Connection reset by peer (error 104)
2026-09-25T00:52:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:35 dist-test-slave-05m0 krb5kdc[22020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:35 dist-test-slave-05m0 krb5kdc[22020](info): set up 2 sockets
Sep 25 00:52:35 dist-test-slave-05m0 krb5kdc[22020](info): commencing operation
krb5kdc: starting...
W20260925 00:52:36.421693 997 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:36 dist-test-slave-05m0 krb5kdc[22020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:36.540107 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:36.650063 22033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:36.650384 22033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:36.650468 22033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:36.654624 22033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:36.654721 22033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:36.654759 22033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:36.654815 22033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:36.654856 22033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:36.660128 22033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:36.661176 22033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:36.662257 22033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:36.670477 22038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:36.670536 22041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:36.670670 22033 server_base.cc:1034] running on GCE node
W20260925 00:52:36.670549 22039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:36.671346 22033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:36.672407 22033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:36.673590 22033 hybrid_clock.cc:648] HybridClock initialized: now 1790297556673572 us; error 30 us; skew 500 ppm
Sep 25 00:52:36 dist-test-slave-05m0 krb5kdc[22020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:36.680773 22033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:36.681731 22033 webserver.cc:460] Webserver started at http://127.0.0.1:46507/ using document root <none> and password file <none>
I20260925 00:52:36.682301 22033 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:36.682395 22033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:36.682634 22033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:36.684633 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "b90bc19ddd984249816cca0f8f7d068c"
format_stamp: "Formatted at 2026-09-25 00:52:36 on dist-test-slave-05m0"
server_key: "ecf445282afe22cd7718f5c0f3ad604b"
server_key_iv: "6742090758005be87130e8e2c88c1f4f"
server_key_version: "encryptionkey@0"
I20260925 00:52:36.685187 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "b90bc19ddd984249816cca0f8f7d068c"
format_stamp: "Formatted at 2026-09-25 00:52:36 on dist-test-slave-05m0"
server_key: "ecf445282afe22cd7718f5c0f3ad604b"
server_key_iv: "6742090758005be87130e8e2c88c1f4f"
server_key_version: "encryptionkey@0"
I20260925 00:52:36.688927 22033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 00:52:36.691253 22048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:36.692483 22033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:36.692641 22033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "b90bc19ddd984249816cca0f8f7d068c"
format_stamp: "Formatted at 2026-09-25 00:52:36 on dist-test-slave-05m0"
server_key: "ecf445282afe22cd7718f5c0f3ad604b"
server_key_iv: "6742090758005be87130e8e2c88c1f4f"
server_key_version: "encryptionkey@0"
I20260925 00:52:36.692754 22033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:36.725834 22033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:36.728775 22033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:36.728984 22033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:36.735883 22033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44257
I20260925 00:52:36.735913 22100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44257 every 8 connection(s)
I20260925 00:52:36.736804 22033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:36.739467 22101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:36.744699 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c: Bootstrap starting.
I20260925 00:52:36.746490 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 22033
I20260925 00:52:36.746649 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:36.746928 997 external_mini_cluster.cc:1428] Setting key c6de6f0200d408e75d32dfead9874a61
I20260925 00:52:36.747361 22101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:36.748317 22101 log.cc:824] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:36.750403 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c: No bootstrap required, opened a new log
I20260925 00:52:36.754585 22101 raft_consensus.cc:348] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90bc19ddd984249816cca0f8f7d068c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260925 00:52:36.754832 22101 raft_consensus.cc:374] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:36.754927 22101 raft_consensus.cc:729] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90bc19ddd984249816cca0f8f7d068c, State: Initialized, Role: FOLLOWER
I20260925 00:52:36.755437 22101 consensus_queue.cc:260] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90bc19ddd984249816cca0f8f7d068c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260925 00:52:36.755589 22101 raft_consensus.cc:388] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:36.755676 22101 raft_consensus.cc:482] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:36.755785 22101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:36.756379 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:36.748091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:47233 (local address 127.0.0.1:44257)
0925 00:52:36.748526 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:36.748542 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:36.748576 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:52:36.749377 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:36.749446 (+ 69us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:36.749798 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:36.750072 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:36.750456 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:36.751479 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:36.752730 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:36.753182 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:36.753300 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:36.754267 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:36.754272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:36.754284 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:36.754667 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:36.754728 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:36.754752 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:36.754998 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:36.755386 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 00:52:36.755498 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":145,"threads_started":1}
I20260925 00:52:36.756801 22101 raft_consensus.cc:504] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90bc19ddd984249816cca0f8f7d068c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260925 00:52:36.757140 22101 leader_election.cc:302] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90bc19ddd984249816cca0f8f7d068c; no voters:
I20260925 00:52:36.757468 22101 leader_election.cc:288] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:36.757546 22106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:36.757766 22106 raft_consensus.cc:686] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [term 1 LEADER]: Becoming Leader. State: Replica: b90bc19ddd984249816cca0f8f7d068c, State: Running, Role: LEADER
I20260925 00:52:36.758129 22106 consensus_queue.cc:237] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90bc19ddd984249816cca0f8f7d068c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260925 00:52:36.758769 22101 sys_catalog.cc:564] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:36.759637 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:36.759742 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b90bc19ddd984249816cca0f8f7d068c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90bc19ddd984249816cca0f8f7d068c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260925 00:52:36.759856 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:36.760068 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b90bc19ddd984249816cca0f8f7d068c. Latest consensus state: current_term: 1 leader_uuid: "b90bc19ddd984249816cca0f8f7d068c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90bc19ddd984249816cca0f8f7d068c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260925 00:52:36.760164 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:36.760404 22115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:36.763892 22115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:36.767984 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:36.761895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:43357 (local address 127.0.0.1:44257)
0925 00:52:36.761997 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:36.762003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:36.762293 (+ 290us) server_negotiation.cc:407] Connection header received
0925 00:52:36.762411 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:36.762413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:36.762456 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:36.762537 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:36.762820 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:36.763593 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:36.764679 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:36.765025 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:36.765133 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:36.767449 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:36.767453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:36.767456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:36.767468 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:36.767495 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:36.767512 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:36.767620 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:36.767813 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 00:52:36.767852 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 00:52:36.769302 22115 catalog_manager.cc:1347] Generated new cluster ID: 49e163520ba74fb78a33a076912e1d3d
I20260925 00:52:36.769361 22115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:36.779958 22115 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:36.781077 22115 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:36.787798 22115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b90bc19ddd984249816cca0f8f7d068c: Generated new TSK 0
I20260925 00:52:36.788424 22115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:36.797516 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 22033
W20260925 00:52:36.802670 22018 connection.cc:535] client connection to 127.0.0.1:44257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44257): Connection reset by peer (error 104)
2026-09-25T00:52:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:36 dist-test-slave-05m0 krb5kdc[22132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:36 dist-test-slave-05m0 krb5kdc[22132](info): set up 2 sockets
Sep 25 00:52:36 dist-test-slave-05m0 krb5kdc[22132](info): commencing operation
krb5kdc: starting...
W20260925 00:52:37.881702 997 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:37 dist-test-slave-05m0 krb5kdc[22132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:38.002372 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:38.113775 22145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:38.114035 22145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:38.114079 22145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:38.118355 22145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:38.118422 22145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:38.118444 22145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:38.118500 22145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:38.118562 22145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:38.124112 22145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:38.125186 22145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:38.126245 22145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:52:38.134473 22145 server_base.cc:1034] running on GCE node
W20260925 00:52:38.134449 22153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:38.134456 22151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:38.134759 22150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:38.135289 22145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:38.136339 22145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:38.137461 22145 hybrid_clock.cc:648] HybridClock initialized: now 1790297558137426 us; error 45 us; skew 500 ppm
Sep 25 00:52:38 dist-test-slave-05m0 krb5kdc[22132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:38.140242 22145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:38.141127 22145 webserver.cc:460] Webserver started at http://127.0.0.1:43339/ using document root <none> and password file <none>
I20260925 00:52:38.141742 22145 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:38.141834 22145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:38.142057 22145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:38.144040 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "4fc9f3a6af454157a08fd19d228b4502"
format_stamp: "Formatted at 2026-09-25 00:52:38 on dist-test-slave-05m0"
server_key: "d1759c2f4d43eeeefa65dace0464c8f1"
server_key_iv: "f45f8b3fe058220b511189589f7815c7"
server_key_version: "encryptionkey@0"
I20260925 00:52:38.144639 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "4fc9f3a6af454157a08fd19d228b4502"
format_stamp: "Formatted at 2026-09-25 00:52:38 on dist-test-slave-05m0"
server_key: "d1759c2f4d43eeeefa65dace0464c8f1"
server_key_iv: "f45f8b3fe058220b511189589f7815c7"
server_key_version: "encryptionkey@0"
I20260925 00:52:38.148327 22145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 00:52:38.150760 22160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:38.151963 22145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:52:38.152127 22145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "4fc9f3a6af454157a08fd19d228b4502"
format_stamp: "Formatted at 2026-09-25 00:52:38 on dist-test-slave-05m0"
server_key: "d1759c2f4d43eeeefa65dace0464c8f1"
server_key_iv: "f45f8b3fe058220b511189589f7815c7"
server_key_version: "encryptionkey@0"
I20260925 00:52:38.152242 22145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:38.156715 22145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:38.159994 22145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:38.160198 22145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:38.167133 22145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34919
I20260925 00:52:38.167153 22212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34919 every 8 connection(s)
I20260925 00:52:38.168066 22145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:38.168386 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 22145
I20260925 00:52:38.168557 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:38.168897 997 external_mini_cluster.cc:1428] Setting key fb5fb6056769c4c4d04ff0e42e4ee2db
I20260925 00:52:38.171419 22213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:38.174091 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:38.170271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:48211 (local address 127.0.0.1:34919)
0925 00:52:38.170756 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:38.170771 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:38.170805 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:52:38.171421 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:38.171436 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:38.171772 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:38.172099 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:38.172909 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:38.172915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:38.172936 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:38.172947 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:38.173002 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:38.173020 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:38.173167 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:38.173238 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 00:52:38.173386 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":146,"threads_started":1}
I20260925 00:52:38.176959 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:38.177901 22213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502: Bootstrap starting.
I20260925 00:52:38.180588 22213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:38.180678 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:38.179075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:45677 (local address 127.0.0.1:34919)
0925 00:52:38.179196 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:38.179201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:38.179339 (+ 138us) server_negotiation.cc:407] Connection header received
0925 00:52:38.179409 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:38.179412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:38.179447 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:38.179539 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:38.180241 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:38.180245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:38.180248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:38.180261 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:38.180292 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:38.180307 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:38.180416 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:38.180465 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 00:52:38.180510 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:52:38.181488 22213 log.cc:824] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:38.183384 22213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502: No bootstrap required, opened a new log
I20260925 00:52:38.187007 22213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc9f3a6af454157a08fd19d228b4502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34919 } }
I20260925 00:52:38.187215 22213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:38.187278 22213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc9f3a6af454157a08fd19d228b4502, State: Initialized, Role: FOLLOWER
I20260925 00:52:38.187733 22213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc9f3a6af454157a08fd19d228b4502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34919 } }
I20260925 00:52:38.187882 22213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:38.187964 22213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:38.188079 22213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:38.188935 22213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc9f3a6af454157a08fd19d228b4502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34919 } }
I20260925 00:52:38.189268 22213 leader_election.cc:302] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc9f3a6af454157a08fd19d228b4502; no voters:
I20260925 00:52:38.189589 22213 leader_election.cc:288] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:38.189671 22223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:38.189955 22223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [term 1 LEADER]: Becoming Leader. State: Replica: 4fc9f3a6af454157a08fd19d228b4502, State: Running, Role: LEADER
I20260925 00:52:38.190312 22223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc9f3a6af454157a08fd19d228b4502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34919 } }
I20260925 00:52:38.190797 22213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:38.191862 22224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fc9f3a6af454157a08fd19d228b4502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc9f3a6af454157a08fd19d228b4502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34919 } } }
I20260925 00:52:38.192133 22225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fc9f3a6af454157a08fd19d228b4502. Latest consensus state: current_term: 1 leader_uuid: "4fc9f3a6af454157a08fd19d228b4502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc9f3a6af454157a08fd19d228b4502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34919 } } }
I20260925 00:52:38.192241 22224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:38.192325 22225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:38.193215 22233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:38.195672 22233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:38.200728 22233 catalog_manager.cc:1347] Generated new cluster ID: bef94cfd3e9f4236ba60f288d364c271
I20260925 00:52:38.200789 22233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:38.208333 22233 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:38.208632 22233 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:38.214012 22233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fc9f3a6af454157a08fd19d228b4502: Generated new TSK 0
I20260925 00:52:38.214833 22233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:38.224905 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 22145
2026-09-25T00:52:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:38 dist-test-slave-05m0 krb5kdc[22244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:38 dist-test-slave-05m0 krb5kdc[22244](info): set up 2 sockets
Sep 25 00:52:38 dist-test-slave-05m0 krb5kdc[22244](info): commencing operation
krb5kdc: starting...
W20260925 00:52:39.269727 997 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:39 dist-test-slave-05m0 krb5kdc[22244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:39.394616 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:39.503104 22257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:39.503379 22257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:39.503420 22257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:39.507498 22257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:39.507562 22257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:39.507581 22257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:39.507597 22257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:39.507649 22257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:39.513005 22257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:39.514153 22257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:39.515174 22257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:39.523833 22263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:39.523813 22262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:39.523990 22257 server_base.cc:1034] running on GCE node
W20260925 00:52:39.523818 22265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:39.524694 22257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:39.525784 22257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:39.526954 22257 hybrid_clock.cc:648] HybridClock initialized: now 1790297559526934 us; error 28 us; skew 500 ppm
Sep 25 00:52:39 dist-test-slave-05m0 krb5kdc[22244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:39.529712 22257 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:39.530578 22257 webserver.cc:460] Webserver started at http://127.0.0.1:40659/ using document root <none> and password file <none>
I20260925 00:52:39.531106 22257 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:39.531172 22257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:39.531404 22257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:39.533515 22257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "72b8d90a457f4cb087c57d53d8661e47"
format_stamp: "Formatted at 2026-09-25 00:52:39 on dist-test-slave-05m0"
server_key: "c0aa6feb1471737ca9a20c95446aa7c2"
server_key_iv: "8bff5e60dedc90f561f4510eb8452d2b"
server_key_version: "encryptionkey@0"
I20260925 00:52:39.534072 22257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "72b8d90a457f4cb087c57d53d8661e47"
format_stamp: "Formatted at 2026-09-25 00:52:39 on dist-test-slave-05m0"
server_key: "c0aa6feb1471737ca9a20c95446aa7c2"
server_key_iv: "8bff5e60dedc90f561f4510eb8452d2b"
server_key_version: "encryptionkey@0"
I20260925 00:52:39.537788 22257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:52:39.540206 22272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:39.541646 22257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 00:52:39.541796 22257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "72b8d90a457f4cb087c57d53d8661e47"
format_stamp: "Formatted at 2026-09-25 00:52:39 on dist-test-slave-05m0"
server_key: "c0aa6feb1471737ca9a20c95446aa7c2"
server_key_iv: "8bff5e60dedc90f561f4510eb8452d2b"
server_key_version: "encryptionkey@0"
I20260925 00:52:39.541952 22257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:39.570541 22257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:39.573779 22257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:39.574014 22257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:39.581218 22257 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45755
I20260925 00:52:39.581278 22324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45755 every 8 connection(s)
I20260925 00:52:39.582309 22257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:39.585129 22325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:39.590570 22325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47: Bootstrap starting.
I20260925 00:52:39.590947 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 22257
I20260925 00:52:39.591080 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:39.591398 997 external_mini_cluster.cc:1428] Setting key ea8045c13e5b5956838826bf6e408de8
I20260925 00:52:39.593348 22325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:39.594269 22325 log.cc:824] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:39.596395 22325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47: No bootstrap required, opened a new log
I20260925 00:52:39.600415 22325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8d90a457f4cb087c57d53d8661e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260925 00:52:39.600659 22325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:39.600759 22325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b8d90a457f4cb087c57d53d8661e47, State: Initialized, Role: FOLLOWER
Sep 25 00:52:39 dist-test-slave-05m0 krb5kdc[22244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 00:52:39.601191 22325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8d90a457f4cb087c57d53d8661e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260925 00:52:39.601421 22325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:39.601495 22325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:39.601609 22325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:39.602845 22325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8d90a457f4cb087c57d53d8661e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260925 00:52:39.603183 22325 leader_election.cc:302] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72b8d90a457f4cb087c57d53d8661e47; no voters:
I20260925 00:52:39.603518 22325 leader_election.cc:288] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:39.603588 22330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:39.603802 22330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [term 1 LEADER]: Becoming Leader. State: Replica: 72b8d90a457f4cb087c57d53d8661e47, State: Running, Role: LEADER
I20260925 00:52:39.604138 22330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8d90a457f4cb087c57d53d8661e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260925 00:52:39.604807 22325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:39.605860 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72b8d90a457f4cb087c57d53d8661e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8d90a457f4cb087c57d53d8661e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } } }
I20260925 00:52:39.605997 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:39.606395 22339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:39.606980 22332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b8d90a457f4cb087c57d53d8661e47. Latest consensus state: current_term: 1 leader_uuid: "72b8d90a457f4cb087c57d53d8661e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b8d90a457f4cb087c57d53d8661e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } } }
I20260925 00:52:39.607133 22332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:39.607877 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:39.592933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:43407 (local address 127.0.0.1:45755)
0925 00:52:39.593413 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:39.593436 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:39.593477 (+ 41us) server_negotiation.cc:407] Connection header received
0925 00:52:39.594381 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:39.594405 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:39.594827 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:39.595188 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:39.595893 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:39.597360 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:39.598260 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:39.598976 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:39.599171 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:39.601955 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:39.601961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:39.601976 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:39.602007 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:39.604050 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:39.604692 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:39.604696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:39.604700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:39.604842 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:39.605217 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:39.605220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:39.605222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:39.605839 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:39.606060 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:39.606632 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 00:52:39.606869 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":130,"threads_started":1}
I20260925 00:52:39.609716 22339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:39.611506 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:39.615684 22339 catalog_manager.cc:1347] Generated new cluster ID: 69f59f7020794ce28d05e9ba2fbcdffe
I20260925 00:52:39.615753 22339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:39.622336 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:39.613544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:36013 (local address 127.0.0.1:45755)
0925 00:52:39.613660 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:39.613664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:39.613915 (+ 251us) server_negotiation.cc:407] Connection header received
0925 00:52:39.614091 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:39.614094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:39.614154 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:39.614241 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:39.614947 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:39.615542 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:39.616650 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:39.617005 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:39.617116 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:39.618700 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:39.618704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:39.618707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:39.618731 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:39.620449 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:39.620958 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:39.620962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:39.620964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:39.621047 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:39.621453 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:39.621456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:39.621458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:39.621660 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:39.621809 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:39.622080 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 00:52:39.622184 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:52:39.624753 22339 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:39.625689 22339 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:39.629557 22339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72b8d90a457f4cb087c57d53d8661e47: Generated new TSK 0
I20260925 00:52:39.630288 22339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:39.636068 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 22257
W20260925 00:52:39.641496 22242 connection.cc:535] client connection to 127.0.0.1:45755 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45755): Connection reset by peer (error 104)
2026-09-25T00:52:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:39 dist-test-slave-05m0 krb5kdc[22356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:39 dist-test-slave-05m0 krb5kdc[22356](info): set up 2 sockets
Sep 25 00:52:39 dist-test-slave-05m0 krb5kdc[22356](info): commencing operation
krb5kdc: starting...
W20260925 00:52:40.673666 997 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:40 dist-test-slave-05m0 krb5kdc[22356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:40.796741 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:40.907130 22369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:40.907420 22369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:40.907493 22369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:40.911463 22369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:40.911559 22369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:40.911597 22369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:40.911634 22369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:40.911670 22369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:40.916823 22369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:40.917896 22369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:40.918877 22369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:40.927380 22375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:40.927491 22369 server_base.cc:1034] running on GCE node
W20260925 00:52:40.927439 22374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:40.927722 22377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:40.928323 22369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:40.929435 22369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:40.930609 22369 hybrid_clock.cc:648] HybridClock initialized: now 1790297560930596 us; error 32 us; skew 500 ppm
Sep 25 00:52:40 dist-test-slave-05m0 krb5kdc[22356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:40.933451 22369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:40.934325 22369 webserver.cc:460] Webserver started at http://127.0.0.1:45423/ using document root <none> and password file <none>
I20260925 00:52:40.934859 22369 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:40.934916 22369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:40.935145 22369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:40.937115 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "d8381e5818c14ed6a39accffc65a58ac"
format_stamp: "Formatted at 2026-09-25 00:52:40 on dist-test-slave-05m0"
server_key: "52c1c2b6ed409668bcfc555de4b55d81"
server_key_iv: "3ad6eb167b2ed35dcf44ffbaf5ed7cab"
server_key_version: "encryptionkey@0"
I20260925 00:52:40.937693 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "d8381e5818c14ed6a39accffc65a58ac"
format_stamp: "Formatted at 2026-09-25 00:52:40 on dist-test-slave-05m0"
server_key: "52c1c2b6ed409668bcfc555de4b55d81"
server_key_iv: "3ad6eb167b2ed35dcf44ffbaf5ed7cab"
server_key_version: "encryptionkey@0"
I20260925 00:52:40.941318 22369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:52:40.943775 22384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:40.945012 22369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:40.945137 22369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "d8381e5818c14ed6a39accffc65a58ac"
format_stamp: "Formatted at 2026-09-25 00:52:40 on dist-test-slave-05m0"
server_key: "52c1c2b6ed409668bcfc555de4b55d81"
server_key_iv: "3ad6eb167b2ed35dcf44ffbaf5ed7cab"
server_key_version: "encryptionkey@0"
I20260925 00:52:40.945250 22369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:40.971354 22369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:40.974664 22369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:40.974869 22369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:40.982040 22369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41979
I20260925 00:52:40.982060 22436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41979 every 8 connection(s)
I20260925 00:52:40.982998 22369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:40.983889 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 22369
I20260925 00:52:40.984037 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:40.984398 997 external_mini_cluster.cc:1428] Setting key 78ebe89cc76abc4296d67f77ce9f77ab
I20260925 00:52:40.986323 22437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:40.992508 22437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac: Bootstrap starting.
I20260925 00:52:40.993209 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:40.985733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:58053 (local address 127.0.0.1:41979)
0925 00:52:40.986125 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:40.986150 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:40.986186 (+ 36us) server_negotiation.cc:407] Connection header received
0925 00:52:40.986853 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:40.986869 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:40.987231 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:40.987572 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:40.987891 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:40.988855 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:40.990085 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:40.990538 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:40.990646 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:40.991545 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:40.991550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:40.991561 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:40.991924 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:40.991972 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:40.991992 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:40.992087 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:40.992329 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 00:52:40.992463 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":134,"threads_started":1}
I20260925 00:52:40.994853 22437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:40.995743 22437 log.cc:824] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:40.996163 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:40.998008 22437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac: No bootstrap required, opened a new log
I20260925 00:52:41.001569 22437 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8381e5818c14ed6a39accffc65a58ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260925 00:52:41.001854 22437 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:41.001943 22437 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8381e5818c14ed6a39accffc65a58ac, State: Initialized, Role: FOLLOWER
I20260925 00:52:41.002323 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:40.998421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:46425 (local address 127.0.0.1:41979)
0925 00:52:40.998530 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:40.998534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:40.998722 (+ 188us) server_negotiation.cc:407] Connection header received
0925 00:52:40.998829 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:40.998834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:40.998881 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:40.998963 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:40.999213 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:40.999720 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:41.000523 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:41.000742 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:41.000853 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:41.001759 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:41.001762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:41.001764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:41.001774 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:41.001797 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:41.001806 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:41.001911 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:41.002133 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 00:52:41.002184 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 00:52:41.002441 22437 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8381e5818c14ed6a39accffc65a58ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260925 00:52:41.002615 22437 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:41.002707 22437 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:41.002805 22437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:41.003810 22437 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8381e5818c14ed6a39accffc65a58ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260925 00:52:41.004150 22437 leader_election.cc:302] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8381e5818c14ed6a39accffc65a58ac; no voters:
I20260925 00:52:41.004492 22437 leader_election.cc:288] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:41.004557 22447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:41.004813 22447 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [term 1 LEADER]: Becoming Leader. State: Replica: d8381e5818c14ed6a39accffc65a58ac, State: Running, Role: LEADER
I20260925 00:52:41.005227 22447 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8381e5818c14ed6a39accffc65a58ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260925 00:52:41.005730 22437 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:41.006846 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8381e5818c14ed6a39accffc65a58ac. Latest consensus state: current_term: 1 leader_uuid: "d8381e5818c14ed6a39accffc65a58ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8381e5818c14ed6a39accffc65a58ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } } }
I20260925 00:52:41.006973 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:41.007418 22448 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8381e5818c14ed6a39accffc65a58ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8381e5818c14ed6a39accffc65a58ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } } }
I20260925 00:52:41.007578 22448 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac [sys.catalog]: This master's current role is: LEADER
W20260925 00:52:41.010321 22462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:41.010385 22462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:41.010463 22456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:41.011126 22456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:41.016182 22456 catalog_manager.cc:1347] Generated new cluster ID: dc2303c6e21b40aa81166fe768896dc4
I20260925 00:52:41.016245 22456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:41.025076 22456 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:41.026343 22456 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:41.040972 22456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8381e5818c14ed6a39accffc65a58ac: Generated new TSK 0
I20260925 00:52:41.041862 22456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:41.050932 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 22369
W20260925 00:52:41.056679 22354 connection.cc:535] client connection to 127.0.0.1:41979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41979): Connection reset by peer (error 104)
2026-09-25T00:52:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:41 dist-test-slave-05m0 krb5kdc[22468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:41 dist-test-slave-05m0 krb5kdc[22468](info): set up 2 sockets
Sep 25 00:52:41 dist-test-slave-05m0 krb5kdc[22468](info): commencing operation
krb5kdc: starting...
W20260925 00:52:42.089720 997 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:42 dist-test-slave-05m0 krb5kdc[22468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 00:52:42.209439 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:42.322789 22481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:42.323062 22481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:42.323135 22481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:42.327281 22481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:42.327364 22481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:42.327404 22481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:42.327461 22481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:42.327500 22481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:42.332688 22481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:42.333724 22481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:42.334744 22481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:42.343078 22487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:42.343150 22481 server_base.cc:1034] running on GCE node
W20260925 00:52:42.343073 22486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:42.343083 22489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:42.343925 22481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:42.345070 22481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:42.346258 22481 hybrid_clock.cc:648] HybridClock initialized: now 1790297562346215 us; error 60 us; skew 500 ppm
Sep 25 00:52:42 dist-test-slave-05m0 krb5kdc[22468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:42.353544 22481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 00:52:42.354435 22481 webserver.cc:460] Webserver started at http://127.0.0.1:45657/ using document root <none> and password file <none>
I20260925 00:52:42.355057 22481 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:42.355147 22481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:42.355379 22481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:42.357435 22481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "795ec6fd23c54094aefcf51f4ea39306"
format_stamp: "Formatted at 2026-09-25 00:52:42 on dist-test-slave-05m0"
server_key: "330ff59d56b66ffd28fdbc940191ae0f"
server_key_iv: "1246c45f96f10ef7840ba8dd1505f72e"
server_key_version: "encryptionkey@0"
I20260925 00:52:42.358006 22481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "795ec6fd23c54094aefcf51f4ea39306"
format_stamp: "Formatted at 2026-09-25 00:52:42 on dist-test-slave-05m0"
server_key: "330ff59d56b66ffd28fdbc940191ae0f"
server_key_iv: "1246c45f96f10ef7840ba8dd1505f72e"
server_key_version: "encryptionkey@0"
I20260925 00:52:42.361588 22481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:52:42.363909 22496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:42.365202 22481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:42.365334 22481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "795ec6fd23c54094aefcf51f4ea39306"
format_stamp: "Formatted at 2026-09-25 00:52:42 on dist-test-slave-05m0"
server_key: "330ff59d56b66ffd28fdbc940191ae0f"
server_key_iv: "1246c45f96f10ef7840ba8dd1505f72e"
server_key_version: "encryptionkey@0"
I20260925 00:52:42.365496 22481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:42.369927 22481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:42.373072 22481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:42.373261 22481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:42.380149 22481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43605
I20260925 00:52:42.380245 22548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43605 every 8 connection(s)
I20260925 00:52:42.381201 22481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:42.383966 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:42.385340 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 22481
I20260925 00:52:42.385527 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:42.385834 997 external_mini_cluster.cc:1428] Setting key 1925dfb77c9c45d702d796be2bbb8425
I20260925 00:52:42.390873 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306: Bootstrap starting.
I20260925 00:52:42.391528 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:42.387187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:43889 (local address 127.0.0.1:43605)
0925 00:52:42.387602 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:42.387623 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:42.387668 (+ 45us) server_negotiation.cc:407] Connection header received
0925 00:52:42.388539 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:42.388563 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:42.388915 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:42.389289 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:42.390147 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:42.390154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:42.390170 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:42.390183 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:42.390249 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:42.390278 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:42.390424 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:42.390494 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 00:52:42.390644 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":99,"threads_started":1}
I20260925 00:52:42.393488 22549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:42.394388 22549 log.cc:824] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:42.394411 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:42.396486 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306: No bootstrap required, opened a new log
I20260925 00:52:42.398212 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:42.396487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:44117 (local address 127.0.0.1:43605)
0925 00:52:42.396615 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:42.396619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:42.396811 (+ 192us) server_negotiation.cc:407] Connection header received
0925 00:52:42.396945 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:42.396947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:42.396973 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:42.397036 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:42.397831 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:42.397834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:42.397836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:42.397845 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:42.397868 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:42.397878 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:42.397944 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:42.398069 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 00:52:42.398100 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 00:52:42.400877 22549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ec6fd23c54094aefcf51f4ea39306" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } }
I20260925 00:52:42.401124 22549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:42.401188 22549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795ec6fd23c54094aefcf51f4ea39306, State: Initialized, Role: FOLLOWER
I20260925 00:52:42.401723 22549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ec6fd23c54094aefcf51f4ea39306" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } }
I20260925 00:52:42.401889 22549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:42.401978 22549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:42.402083 22549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:42.403003 22549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ec6fd23c54094aefcf51f4ea39306" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } }
I20260925 00:52:42.403349 22549 leader_election.cc:302] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 795ec6fd23c54094aefcf51f4ea39306; no voters:
I20260925 00:52:42.403713 22549 leader_election.cc:288] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:42.403815 22559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:42.404047 22559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [term 1 LEADER]: Becoming Leader. State: Replica: 795ec6fd23c54094aefcf51f4ea39306, State: Running, Role: LEADER
I20260925 00:52:42.404410 22559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ec6fd23c54094aefcf51f4ea39306" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } }
I20260925 00:52:42.404975 22549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:42.406750 22560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "795ec6fd23c54094aefcf51f4ea39306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ec6fd23c54094aefcf51f4ea39306" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } } }
I20260925 00:52:42.406919 22560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:42.407164 22561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 795ec6fd23c54094aefcf51f4ea39306. Latest consensus state: current_term: 1 leader_uuid: "795ec6fd23c54094aefcf51f4ea39306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ec6fd23c54094aefcf51f4ea39306" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43605 } } }
I20260925 00:52:42.407292 22561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306 [sys.catalog]: This master's current role is: LEADER
W20260925 00:52:42.409773 22575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:42.409852 22575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:42.409950 22574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:42.410955 22574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:42.417496 22574 catalog_manager.cc:1347] Generated new cluster ID: d33d462eeaf342968265ce75ecbdda23
I20260925 00:52:42.417610 22574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:42.428789 22574 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:42.429080 22574 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:42.442080 22574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 795ec6fd23c54094aefcf51f4ea39306: Generated new TSK 0
I20260925 00:52:42.442793 22574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:42.455118 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 22481
2026-09-25T00:52:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:42 dist-test-slave-05m0 krb5kdc[22580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:42 dist-test-slave-05m0 krb5kdc[22580](info): set up 2 sockets
Sep 25 00:52:42 dist-test-slave-05m0 krb5kdc[22580](info): commencing operation
krb5kdc: starting...
W20260925 00:52:43.522253 997 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:43 dist-test-slave-05m0 krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:43.644567 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:42669
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:42669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:43.754346 22593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:43.754663 22593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:43.754740 22593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:43.758785 22593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:43.758878 22593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:43.758916 22593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:43.758972 22593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:43.759009 22593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:43.764359 22593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:42669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:42669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:43.765422 22593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:43.766444 22593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:43.775167 22599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:43.775126 22593 server_base.cc:1034] running on GCE node
W20260925 00:52:43.775126 22598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:43.775132 22601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:43.775843 22593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:43.776939 22593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:43.778129 22593 hybrid_clock.cc:648] HybridClock initialized: now 1790297563778110 us; error 30 us; skew 500 ppm
Sep 25 00:52:43 dist-test-slave-05m0 krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:43.780750 22593 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:43.781636 22593 webserver.cc:460] Webserver started at http://127.0.249.126:42615/ using document root <none> and password file <none>
I20260925 00:52:43.782172 22593 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:43.782279 22593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:43.782506 22593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:43.784485 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "7f1622d73132489ead4717a2eade7cb1"
format_stamp: "Formatted at 2026-09-25 00:52:43 on dist-test-slave-05m0"
server_key: "f753bac66ef88cbcb2d34c34990c0fbe"
server_key_iv: "83e4dbf9e3d0837c70483301ac565fba"
server_key_version: "encryptionkey@0"
I20260925 00:52:43.785043 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "7f1622d73132489ead4717a2eade7cb1"
format_stamp: "Formatted at 2026-09-25 00:52:43 on dist-test-slave-05m0"
server_key: "f753bac66ef88cbcb2d34c34990c0fbe"
server_key_iv: "83e4dbf9e3d0837c70483301ac565fba"
server_key_version: "encryptionkey@0"
I20260925 00:52:43.788630 22593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 00:52:43.791018 22608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:43.792179 22593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:52:43.792322 22593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "7f1622d73132489ead4717a2eade7cb1"
format_stamp: "Formatted at 2026-09-25 00:52:43 on dist-test-slave-05m0"
server_key: "f753bac66ef88cbcb2d34c34990c0fbe"
server_key_iv: "83e4dbf9e3d0837c70483301ac565fba"
server_key_version: "encryptionkey@0"
I20260925 00:52:43.792433 22593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:43.815136 22593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:43.818028 22593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:43.818240 22593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:43.825471 22593 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:42669
I20260925 00:52:43.825505 22660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:42669 every 8 connection(s)
I20260925 00:52:43.826448 22593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:43.829275 22661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:43.831017 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 22593
I20260925 00:52:43.831151 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:43.831423 997 external_mini_cluster.cc:1428] Setting key dd7990ec44d2a69698f9661eb3262594
I20260925 00:52:43.835944 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1: Bootstrap starting.
I20260925 00:52:43.838426 22661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:43.839272 22661 log.cc:824] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:52:43 dist-test-slave-05m0 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:43.841156 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1: No bootstrap required, opened a new log
I20260925 00:52:43.845968 22661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1622d73132489ead4717a2eade7cb1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42669 } }
I20260925 00:52:43.846225 22661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:43.846272 22661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1622d73132489ead4717a2eade7cb1, State: Initialized, Role: FOLLOWER
I20260925 00:52:43.846421 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:43.832768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44929 (local address 127.0.249.126:42669)
0925 00:52:43.833176 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:43.833194 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:43.833236 (+ 42us) server_negotiation.cc:407] Connection header received
0925 00:52:43.834097 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:43.834122 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:43.834544 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:43.834893 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:43.835571 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:43.837003 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:43.837947 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:43.838665 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:43.838827 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:43.841295 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:43.841301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:43.841317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:43.841346 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:43.843340 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:43.843836 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:43.843842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:43.843848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:43.843986 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:43.844281 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:43.844285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:43.844287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:43.844771 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:43.844990 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:43.845215 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 00:52:43.845479 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260925 00:52:43.846748 22661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1622d73132489ead4717a2eade7cb1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42669 } }
I20260925 00:52:43.846906 22661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:43.846976 22661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:43.847074 22661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:43.847918 22661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1622d73132489ead4717a2eade7cb1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42669 } }
I20260925 00:52:43.848292 22661 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1622d73132489ead4717a2eade7cb1; no voters:
I20260925 00:52:43.848675 22661 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:43.848783 22666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:43.849036 22666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [term 1 LEADER]: Becoming Leader. State: Replica: 7f1622d73132489ead4717a2eade7cb1, State: Running, Role: LEADER
I20260925 00:52:43.849489 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:43.849443 22666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1622d73132489ead4717a2eade7cb1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42669 } }
I20260925 00:52:43.849942 22661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:43.851351 22668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1622d73132489ead4717a2eade7cb1. Latest consensus state: current_term: 1 leader_uuid: "7f1622d73132489ead4717a2eade7cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1622d73132489ead4717a2eade7cb1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42669 } } }
I20260925 00:52:43.851486 22668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:43.852409 22667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f1622d73132489ead4717a2eade7cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1622d73132489ead4717a2eade7cb1" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 42669 } } }
I20260925 00:52:43.852519 22667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:43.852804 22684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:43.855937 22684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:43.861557 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:43.852366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42297 (local address 127.0.249.126:42669)
0925 00:52:43.852466 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:43.852471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:43.852878 (+ 407us) server_negotiation.cc:407] Connection header received
0925 00:52:43.853041 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:43.853044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:43.853088 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:43.853197 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:43.853941 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:43.854676 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:43.855762 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:43.856097 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:43.856209 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:43.858062 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:43.858065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:43.858067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:43.858086 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:43.859504 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:43.860018 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:43.860021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:43.860023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:43.860102 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:43.860432 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:43.860435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:43.860437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:43.860598 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:43.860752 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:43.861293 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 00:52:43.861426 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 00:52:43.863245 22684 catalog_manager.cc:1347] Generated new cluster ID: 6ce6bc9ffdd94a43bc4f4da7339d9c91
I20260925 00:52:43.863317 22684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:43.876070 22684 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:43.877439 22684 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:43.886391 22684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f1622d73132489ead4717a2eade7cb1: Generated new TSK 0
I20260925 00:52:43.887094 22684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:43.899183 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 22593
W20260925 00:52:43.904697 22578 connection.cc:535] client connection to 127.0.249.126:42669 recv error: Network error: failed to read from TLS socket (remote: 127.0.249.126:42669): Connection reset by peer (error 104)
2026-09-25T00:52:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:43 dist-test-slave-05m0 krb5kdc[22692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:43 dist-test-slave-05m0 krb5kdc[22692](info): set up 2 sockets
Sep 25 00:52:43 dist-test-slave-05m0 krb5kdc[22692](info): commencing operation
krb5kdc: starting...
W20260925 00:52:44.949586 997 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:45 dist-test-slave-05m0 krb5kdc[22692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:45.072611 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:37237
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:37237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:45.182106 22705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:45.182354 22705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:45.182395 22705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:45.186404 22705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:45.186463 22705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:45.186483 22705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:45.186499 22705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:45.186514 22705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:45.191860 22705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:37237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:37237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:45.192799 22705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:45.193742 22705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:45.201756 22710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:45.201768 22711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:45.201781 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:45.202018 22705 server_base.cc:1034] running on GCE node
I20260925 00:52:45.202561 22705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:45.203646 22705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:45.204843 22705 hybrid_clock.cc:648] HybridClock initialized: now 1790297565204834 us; error 30 us; skew 500 ppm
Sep 25 00:52:45 dist-test-slave-05m0 krb5kdc[22692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:45.207543 22705 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:45.208459 22705 webserver.cc:460] Webserver started at http://127.0.249.126:43043/ using document root <none> and password file <none>
I20260925 00:52:45.209009 22705 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:45.209092 22705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:45.209300 22705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:45.211315 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "8fe54c5cf51d4f4fb4a7318eb788f581"
format_stamp: "Formatted at 2026-09-25 00:52:45 on dist-test-slave-05m0"
server_key: "fc9674ab73e78eb5dfbe800a2bacd1be"
server_key_iv: "82c81fad842c2433f0c7e858e9f6eaf9"
server_key_version: "encryptionkey@0"
I20260925 00:52:45.211874 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "8fe54c5cf51d4f4fb4a7318eb788f581"
format_stamp: "Formatted at 2026-09-25 00:52:45 on dist-test-slave-05m0"
server_key: "fc9674ab73e78eb5dfbe800a2bacd1be"
server_key_iv: "82c81fad842c2433f0c7e858e9f6eaf9"
server_key_version: "encryptionkey@0"
I20260925 00:52:45.216033 22705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 00:52:45.218528 22720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:45.219756 22705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 00:52:45.219900 22705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "8fe54c5cf51d4f4fb4a7318eb788f581"
format_stamp: "Formatted at 2026-09-25 00:52:45 on dist-test-slave-05m0"
server_key: "fc9674ab73e78eb5dfbe800a2bacd1be"
server_key_iv: "82c81fad842c2433f0c7e858e9f6eaf9"
server_key_version: "encryptionkey@0"
I20260925 00:52:45.220013 22705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:45.246289 22705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:45.249495 22705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:45.249727 22705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:45.256807 22705 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:37237
I20260925 00:52:45.256830 22772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:37237 every 8 connection(s)
I20260925 00:52:45.257768 22705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:45.258704 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 22705
I20260925 00:52:45.258850 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:45.259148 997 external_mini_cluster.cc:1428] Setting key d6bc5e8159cda49ff594aa200186fb94
I20260925 00:52:45.261013 22773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:45.266566 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581: Bootstrap starting.
I20260925 00:52:45.267967 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:45.260547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48129 (local address 127.0.249.126:37237)
0925 00:52:45.260995 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:45.261010 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:45.261044 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:52:45.261777 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:45.261795 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:45.262123 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:45.262461 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:45.262854 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:45.263816 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:45.264728 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:45.265190 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:45.265368 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:45.266327 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:45.266332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:45.266344 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:45.266761 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:45.266814 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:45.266836 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:45.266992 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:45.267164 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 00:52:45.267300 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":148,"threads_started":1}
I20260925 00:52:45.268833 22773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:45.269726 22773 log.cc:824] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:45.270964 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:45.271725 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581: No bootstrap required, opened a new log
I20260925 00:52:45.275323 22773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe54c5cf51d4f4fb4a7318eb788f581" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37237 } }
I20260925 00:52:45.275548 22773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:45.275660 22773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe54c5cf51d4f4fb4a7318eb788f581, State: Initialized, Role: FOLLOWER
I20260925 00:52:45.276153 22773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe54c5cf51d4f4fb4a7318eb788f581" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37237 } }
I20260925 00:52:45.276328 22773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:45.276422 22773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:45.276520 22773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:45.277010 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:45.273063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50953 (local address 127.0.249.126:37237)
0925 00:52:45.273173 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:45.273177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:45.273349 (+ 172us) server_negotiation.cc:407] Connection header received
0925 00:52:45.273491 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:45.273494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:45.273539 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:45.273622 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:45.273912 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:45.274417 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:45.275227 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:45.275460 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:45.275571 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:45.276464 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:45.276468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:45.276470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:45.276483 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:45.276508 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:45.276520 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:45.276579 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:45.276840 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 00:52:45.276878 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:52:45.277633 22773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe54c5cf51d4f4fb4a7318eb788f581" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37237 } }
I20260925 00:52:45.277993 22773 leader_election.cc:302] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fe54c5cf51d4f4fb4a7318eb788f581; no voters:
I20260925 00:52:45.278316 22773 leader_election.cc:288] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:45.278388 22783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:45.278685 22783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [term 1 LEADER]: Becoming Leader. State: Replica: 8fe54c5cf51d4f4fb4a7318eb788f581, State: Running, Role: LEADER
I20260925 00:52:45.279011 22783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe54c5cf51d4f4fb4a7318eb788f581" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37237 } }
I20260925 00:52:45.279486 22773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:45.280577 22784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fe54c5cf51d4f4fb4a7318eb788f581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe54c5cf51d4f4fb4a7318eb788f581" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37237 } } }
I20260925 00:52:45.280856 22785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fe54c5cf51d4f4fb4a7318eb788f581. Latest consensus state: current_term: 1 leader_uuid: "8fe54c5cf51d4f4fb4a7318eb788f581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe54c5cf51d4f4fb4a7318eb788f581" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 37237 } } }
I20260925 00:52:45.280961 22784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:45.281044 22785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:45.281605 22796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:45.284153 22796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:45.289160 22796 catalog_manager.cc:1347] Generated new cluster ID: 06dde1e53d91497a85a24c389a1351f0
I20260925 00:52:45.289222 22796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:45.306967 22796 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:45.308203 22796 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:45.314059 22796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fe54c5cf51d4f4fb4a7318eb788f581: Generated new TSK 0
I20260925 00:52:45.314810 22796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:45.319278 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 22705
W20260925 00:52:45.324753 22690 connection.cc:535] client connection to 127.0.249.126:37237 recv error: Network error: failed to read from TLS socket (remote: 127.0.249.126:37237): Connection reset by peer (error 104)
2026-09-25T00:52:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:45 dist-test-slave-05m0 krb5kdc[22804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:45 dist-test-slave-05m0 krb5kdc[22804](info): set up 2 sockets
Sep 25 00:52:45 dist-test-slave-05m0 krb5kdc[22804](info): commencing operation
krb5kdc: starting...
W20260925 00:52:46.345983 997 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:46 dist-test-slave-05m0 krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:46.468724 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:35091
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:43531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:35091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:46.575871 22817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:46.576110 22817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:46.576149 22817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:46.580060 22817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:46.580116 22817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:46.580135 22817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:46.580196 22817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:46.580237 22817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:46.585372 22817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:43531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:35091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:35091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:46.586390 22817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:46.587363 22817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:46.596266 22825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:46.596416 22817 server_base.cc:1034] running on GCE node
W20260925 00:52:46.596274 22822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:46.596297 22823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:46.597067 22817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:46.598205 22817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:46.599368 22817 hybrid_clock.cc:648] HybridClock initialized: now 1790297566599352 us; error 29 us; skew 500 ppm
Sep 25 00:52:46 dist-test-slave-05m0 krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:46.602108 22817 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:46.602974 22817 webserver.cc:460] Webserver started at http://127.0.249.126:38171/ using document root <none> and password file <none>
I20260925 00:52:46.603519 22817 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:46.603607 22817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:46.603849 22817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:46.605890 22817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "36849e16151d482ab0e9149da13f4698"
format_stamp: "Formatted at 2026-09-25 00:52:46 on dist-test-slave-05m0"
server_key: "38462846d337ffb4b829ef774f5c7c65"
server_key_iv: "326b67a37e0cd2c545e233d243d5eb04"
server_key_version: "encryptionkey@0"
I20260925 00:52:46.606432 22817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "36849e16151d482ab0e9149da13f4698"
format_stamp: "Formatted at 2026-09-25 00:52:46 on dist-test-slave-05m0"
server_key: "38462846d337ffb4b829ef774f5c7c65"
server_key_iv: "326b67a37e0cd2c545e233d243d5eb04"
server_key_version: "encryptionkey@0"
I20260925 00:52:46.609938 22817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:52:46.612239 22832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:46.613364 22817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:46.613564 22817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "36849e16151d482ab0e9149da13f4698"
format_stamp: "Formatted at 2026-09-25 00:52:46 on dist-test-slave-05m0"
server_key: "38462846d337ffb4b829ef774f5c7c65"
server_key_iv: "326b67a37e0cd2c545e233d243d5eb04"
server_key_version: "encryptionkey@0"
I20260925 00:52:46.613673 22817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:46.618000 22817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:46.620738 22817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:46.620918 22817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:46.627833 22817 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:35091
I20260925 00:52:46.627861 22884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:35091 every 8 connection(s)
I20260925 00:52:46.628695 22817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:46.631328 22885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:46.634603 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 22817
I20260925 00:52:46.634763 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:46.635090 997 external_mini_cluster.cc:1428] Setting key 126c026cf91dd59e9203c55d6576564f
I20260925 00:52:46.637207 22885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698: Bootstrap starting.
I20260925 00:52:46.639884 22885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:46.640681 22885 log.cc:824] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:46.642043 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:46.636561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48993 (local address 127.0.249.126:35091)
0925 00:52:46.637038 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:46.637053 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:46.637123 (+ 70us) server_negotiation.cc:407] Connection header received
0925 00:52:46.637917 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:46.637937 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:46.638410 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:46.638922 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:46.640606 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:46.640612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:46.640627 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:46.640639 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:46.640721 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:46.640754 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:46.640878 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:46.641063 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 00:52:46.641210 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":141,"threads_started":1}
I20260925 00:52:46.643040 22885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698: No bootstrap required, opened a new log
I20260925 00:52:46.644768 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:46.646910 22885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36849e16151d482ab0e9149da13f4698" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35091 } }
I20260925 00:52:46.647117 22885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:46.647177 22885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36849e16151d482ab0e9149da13f4698, State: Initialized, Role: FOLLOWER
I20260925 00:52:46.647620 22885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36849e16151d482ab0e9149da13f4698" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35091 } }
I20260925 00:52:46.647807 22885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:46.647888 22885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:46.647992 22885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:46.648566 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:46.646814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58841 (local address 127.0.249.126:35091)
0925 00:52:46.646937 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:46.646942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:46.647233 (+ 291us) server_negotiation.cc:407] Connection header received
0925 00:52:46.647322 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:46.647325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:46.647359 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:46.647445 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:46.648210 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:46.648217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:46.648223 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:46.648236 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:46.648262 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:46.648277 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:46.648373 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:46.648422 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 00:52:46.648455 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:52:46.649072 22885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36849e16151d482ab0e9149da13f4698" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35091 } }
I20260925 00:52:46.649456 22885 leader_election.cc:302] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36849e16151d482ab0e9149da13f4698; no voters:
I20260925 00:52:46.649786 22885 leader_election.cc:288] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:46.649868 22895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:46.650128 22895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [term 1 LEADER]: Becoming Leader. State: Replica: 36849e16151d482ab0e9149da13f4698, State: Running, Role: LEADER
I20260925 00:52:46.650475 22895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36849e16151d482ab0e9149da13f4698" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35091 } }
I20260925 00:52:46.650991 22885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:46.652070 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36849e16151d482ab0e9149da13f4698. Latest consensus state: current_term: 1 leader_uuid: "36849e16151d482ab0e9149da13f4698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36849e16151d482ab0e9149da13f4698" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35091 } } }
I20260925 00:52:46.652185 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:46.652578 22904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:46.652565 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36849e16151d482ab0e9149da13f4698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36849e16151d482ab0e9149da13f4698" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 35091 } } }
I20260925 00:52:46.653232 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:46.655781 22904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:46.661173 22904 catalog_manager.cc:1347] Generated new cluster ID: 84efdd89e9304459beb1cd1e49858b68
I20260925 00:52:46.661247 22904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:46.668836 22904 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:46.669155 22904 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:46.673558 22904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36849e16151d482ab0e9149da13f4698: Generated new TSK 0
I20260925 00:52:46.674312 22904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:46.683786 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 22817
2026-09-25T00:52:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:46 dist-test-slave-05m0 krb5kdc[22916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:46 dist-test-slave-05m0 krb5kdc[22916](info): set up 2 sockets
Sep 25 00:52:46 dist-test-slave-05m0 krb5kdc[22916](info): commencing operation
krb5kdc: starting...
W20260925 00:52:47.746157 997 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:47 dist-test-slave-05m0 krb5kdc[22916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:47.870678 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36115
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:38427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:36115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:47.979779 22929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:47.980034 22929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:47.980074 22929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:47.983939 22929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:47.983989 22929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:47.984007 22929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:47.984023 22929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:47.984038 22929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:47.989084 22929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:38427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:36115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:47.990136 22929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:47.991089 22929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:47.999481 22935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:47.999480 22937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:47.999501 22934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:48.000020 22929 server_base.cc:1034] running on GCE node
I20260925 00:52:48.000541 22929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:48.001744 22929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:48.002915 22929 hybrid_clock.cc:648] HybridClock initialized: now 1790297568002881 us; error 45 us; skew 500 ppm
Sep 25 00:52:48 dist-test-slave-05m0 krb5kdc[22916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:48.009217 22929 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:48.010227 22929 webserver.cc:460] Webserver started at http://127.0.249.126:37235/ using document root <none> and password file <none>
I20260925 00:52:48.010818 22929 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:48.010912 22929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:48.011139 22929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:48.013137 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "850589aafda840599171dc7dd0b0eb76"
format_stamp: "Formatted at 2026-09-25 00:52:48 on dist-test-slave-05m0"
server_key: "b883f4d454ef651a291321a4db562742"
server_key_iv: "23e94028220bcd2463f05f3be6e4dbdc"
server_key_version: "encryptionkey@0"
I20260925 00:52:48.013751 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "850589aafda840599171dc7dd0b0eb76"
format_stamp: "Formatted at 2026-09-25 00:52:48 on dist-test-slave-05m0"
server_key: "b883f4d454ef651a291321a4db562742"
server_key_iv: "23e94028220bcd2463f05f3be6e4dbdc"
server_key_version: "encryptionkey@0"
I20260925 00:52:48.017369 22929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 00:52:48.019852 22944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:48.021082 22929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:48.021240 22929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "850589aafda840599171dc7dd0b0eb76"
format_stamp: "Formatted at 2026-09-25 00:52:48 on dist-test-slave-05m0"
server_key: "b883f4d454ef651a291321a4db562742"
server_key_iv: "23e94028220bcd2463f05f3be6e4dbdc"
server_key_version: "encryptionkey@0"
I20260925 00:52:48.021360 22929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:48.040558 22929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:48.043447 22929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:48.043658 22929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:48.050578 22929 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:36115
I20260925 00:52:48.050637 22996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:36115 every 8 connection(s)
I20260925 00:52:48.051517 22929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:48.054275 22997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:48.056514 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 22929
I20260925 00:52:48.056659 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:48.056941 997 external_mini_cluster.cc:1428] Setting key 92a9defe7ec54f3003390b8ef17c0d68
I20260925 00:52:48.060235 22997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76: Bootstrap starting.
I20260925 00:52:48.062422 22997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:48.063155 22997 log.cc:824] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:48.065505 22997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76: No bootstrap required, opened a new log
Sep 25 00:52:48 dist-test-slave-05m0 krb5kdc[22916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:48.069075 22997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850589aafda840599171dc7dd0b0eb76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36115 } }
I20260925 00:52:48.069360 22997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:48.069456 22997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850589aafda840599171dc7dd0b0eb76, State: Initialized, Role: FOLLOWER
I20260925 00:52:48.069943 22997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850589aafda840599171dc7dd0b0eb76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36115 } }
I20260925 00:52:48.070096 22997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:48.070179 22997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:48.070276 22997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:48.071221 22997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850589aafda840599171dc7dd0b0eb76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36115 } }
I20260925 00:52:48.071571 22997 leader_election.cc:302] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 850589aafda840599171dc7dd0b0eb76; no voters:
I20260925 00:52:48.071919 22997 leader_election.cc:288] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:48.071995 23002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:48.072201 23002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [term 1 LEADER]: Becoming Leader. State: Replica: 850589aafda840599171dc7dd0b0eb76, State: Running, Role: LEADER
I20260925 00:52:48.072443 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:48.058395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35961 (local address 127.0.249.126:36115)
0925 00:52:48.058818 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:48.058832 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:48.058866 (+ 34us) server_negotiation.cc:407] Connection header received
0925 00:52:48.059484 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:48.059501 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:48.059845 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:48.060176 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:48.060993 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:48.062049 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:48.063016 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:48.063587 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:48.063778 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:48.066700 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:48.066706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:48.066718 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:48.066745 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:48.068890 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:48.069467 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:48.069475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:48.069482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:48.069674 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:48.069964 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:48.069969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:48.069970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:48.070494 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:48.070709 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:48.070961 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 00:52:48.071250 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":123,"threads_started":1}
I20260925 00:52:48.072559 23002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850589aafda840599171dc7dd0b0eb76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36115 } }
I20260925 00:52:48.073206 22997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:48.074352 23004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 850589aafda840599171dc7dd0b0eb76. Latest consensus state: current_term: 1 leader_uuid: "850589aafda840599171dc7dd0b0eb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850589aafda840599171dc7dd0b0eb76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36115 } } }
I20260925 00:52:48.075023 23004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:48.074913 23003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "850589aafda840599171dc7dd0b0eb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850589aafda840599171dc7dd0b0eb76" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 36115 } } }
I20260925 00:52:48.075361 23003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:48.075964 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 00:52:48.078533 23016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:48.078608 23016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:48.078724 23017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:48.079681 23017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:48.085649 23017 catalog_manager.cc:1347] Generated new cluster ID: aac20a91797a4bf099bc0ab3db0c6e3c
I20260925 00:52:48.085711 23017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:48.086841 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:48.078158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39933 (local address 127.0.249.126:36115)
0925 00:52:48.078272 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:48.078278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:48.078546 (+ 268us) server_negotiation.cc:407] Connection header received
0925 00:52:48.078734 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:48.078737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:48.078783 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:48.078860 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:48.079673 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:48.080485 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:48.081621 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:48.081854 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:48.081968 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:48.083318 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:48.083322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:48.083324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:48.083349 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:48.084987 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:48.085490 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:48.085495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:48.085497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:48.085578 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:48.085907 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:48.085911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:48.085912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:48.086139 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:48.086281 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:48.086553 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 00:52:48.086680 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 00:52:48.094420 23017 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:48.095319 23017 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:48.105968 23017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 850589aafda840599171dc7dd0b0eb76: Generated new TSK 0
I20260925 00:52:48.106657 23017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:48.115588 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 22929
2026-09-25T00:52:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:48 dist-test-slave-05m0 krb5kdc[23028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:48 dist-test-slave-05m0 krb5kdc[23028](info): set up 2 sockets
Sep 25 00:52:48 dist-test-slave-05m0 krb5kdc[23028](info): commencing operation
krb5kdc: starting...
W20260925 00:52:49.234316 997 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:49 dist-test-slave-05m0 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:49.360291 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:41861
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:41861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:49.468711 23041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:49.468966 23041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:49.469007 23041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:49.473001 23041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:49.473055 23041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:49.473075 23041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:49.473091 23041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:49.473140 23041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:49.478374 23041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:41861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:41861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:49.479424 23041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:49.480463 23041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:49.488852 23049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:49.488857 23046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:49.488891 23047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:49.489120 23041 server_base.cc:1034] running on GCE node
I20260925 00:52:49.489753 23041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:49.490834 23041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:49.492023 23041 hybrid_clock.cc:648] HybridClock initialized: now 1790297569492003 us; error 30 us; skew 500 ppm
Sep 25 00:52:49 dist-test-slave-05m0 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:49.494935 23041 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:49.495796 23041 webserver.cc:460] Webserver started at http://127.0.249.126:42827/ using document root <none> and password file <none>
I20260925 00:52:49.496347 23041 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:49.496441 23041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:49.496675 23041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:49.498725 23041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "08f83d81985a458083060c378141cad7"
format_stamp: "Formatted at 2026-09-25 00:52:49 on dist-test-slave-05m0"
server_key: "82e7d0df7b2248d8546f172f8c3804b9"
server_key_iv: "70c6c5b27b85158f17e236da8cd8268f"
server_key_version: "encryptionkey@0"
I20260925 00:52:49.499294 23041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "08f83d81985a458083060c378141cad7"
format_stamp: "Formatted at 2026-09-25 00:52:49 on dist-test-slave-05m0"
server_key: "82e7d0df7b2248d8546f172f8c3804b9"
server_key_iv: "70c6c5b27b85158f17e236da8cd8268f"
server_key_version: "encryptionkey@0"
I20260925 00:52:49.502948 23041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 00:52:49.505307 23056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:49.506599 23041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:49.506770 23041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "08f83d81985a458083060c378141cad7"
format_stamp: "Formatted at 2026-09-25 00:52:49 on dist-test-slave-05m0"
server_key: "82e7d0df7b2248d8546f172f8c3804b9"
server_key_iv: "70c6c5b27b85158f17e236da8cd8268f"
server_key_version: "encryptionkey@0"
I20260925 00:52:49.506891 23041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:49.520629 23041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:49.523393 23041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:49.523603 23041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:49.530388 23041 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:41861
I20260925 00:52:49.530452 23108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:41861 every 8 connection(s)
I20260925 00:52:49.531368 23041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:49.534113 23109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:49.536294 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 23041
I20260925 00:52:49.536432 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:49.536706 997 external_mini_cluster.cc:1428] Setting key a8cdfaf5510862f27e453d05a6122e93
I20260925 00:52:49.539903 23109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7: Bootstrap starting.
I20260925 00:52:49.542343 23109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:49.543172 23109 log.cc:824] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:49.545356 23109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7: No bootstrap required, opened a new log
I20260925 00:52:49.545727 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:49.537983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44691 (local address 127.0.249.126:41861)
0925 00:52:49.538371 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:49.538393 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:49.538428 (+ 35us) server_negotiation.cc:407] Connection header received
0925 00:52:49.539050 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:49.539066 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:49.539403 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:49.539686 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:49.540095 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:49.541078 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:49.542286 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:49.542798 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:49.542918 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:49.543878 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:49.543887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:49.543905 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:49.544358 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:49.544410 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:49.544434 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:49.544511 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:49.544701 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 00:52:49.544932 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":127,"threads_started":1}
I20260925 00:52:49.548655 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:49.549584 23109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f83d81985a458083060c378141cad7" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41861 } }
I20260925 00:52:49.549851 23109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:49.549918 23109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f83d81985a458083060c378141cad7, State: Initialized, Role: FOLLOWER
I20260925 00:52:49.550443 23109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f83d81985a458083060c378141cad7" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41861 } }
I20260925 00:52:49.550611 23109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:49.550686 23109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:49.550810 23109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:49.551999 23109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f83d81985a458083060c378141cad7" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41861 } }
I20260925 00:52:49.552341 23109 leader_election.cc:302] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08f83d81985a458083060c378141cad7; no voters:
I20260925 00:52:49.552673 23109 leader_election.cc:288] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:49.552789 23119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:49.553035 23119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [term 1 LEADER]: Becoming Leader. State: Replica: 08f83d81985a458083060c378141cad7, State: Running, Role: LEADER
I20260925 00:52:49.553433 23119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f83d81985a458083060c378141cad7" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41861 } }
I20260925 00:52:49.554004 23109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:49.555040 23121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08f83d81985a458083060c378141cad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f83d81985a458083060c378141cad7" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41861 } } }
I20260925 00:52:49.555166 23121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:49.555212 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:49.551203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35585 (local address 127.0.249.126:41861)
0925 00:52:49.551307 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:49.551311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:49.551329 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:52:49.551360 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:49.551362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:49.551393 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:49.551455 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:49.551741 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:49.552313 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:49.553430 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:49.553752 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:49.553859 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:49.554682 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:49.554686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:49.554691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:49.554707 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:49.554748 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:49.554764 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:49.554817 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:49.555020 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 00:52:49.555077 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 00:52:49.555403 23121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08f83d81985a458083060c378141cad7. Latest consensus state: current_term: 1 leader_uuid: "08f83d81985a458083060c378141cad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f83d81985a458083060c378141cad7" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 41861 } } }
I20260925 00:52:49.555476 23121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7 [sys.catalog]: This master's current role is: LEADER
W20260925 00:52:49.558773 23134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:49.558871 23134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:49.558943 23128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:49.559614 23128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:49.565186 23128 catalog_manager.cc:1347] Generated new cluster ID: 5bf8c207bafa4e468fdbc857824e7e1f
I20260925 00:52:49.565263 23128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:49.596738 23128 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:49.597800 23128 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:49.606812 23128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08f83d81985a458083060c378141cad7: Generated new TSK 0
I20260925 00:52:49.607542 23128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:49.614984 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 23041
2026-09-25T00:52:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1499 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:49 dist-test-slave-05m0 krb5kdc[23140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:49 dist-test-slave-05m0 krb5kdc[23140](info): set up 2 sockets
Sep 25 00:52:49 dist-test-slave-05m0 krb5kdc[23140](info): commencing operation
krb5kdc: starting...
W20260925 00:52:50.665827 997 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:50 dist-test-slave-05m0 krb5kdc[23140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:50.786381 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46645
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:46645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:50.895540 23153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:50.895964 23153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:50.896049 23153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:50.900532 23153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:50.900595 23153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:50.900615 23153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:50.900667 23153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:50.900710 23153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:50.906031 23153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:46645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:46645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:50.907050 23153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:50.908041 23153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:50.916447 23159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:50.916471 23161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:50.916443 23158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:50.916851 23153 server_base.cc:1034] running on GCE node
I20260925 00:52:50.917379 23153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:50.918445 23153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:50.919598 23153 hybrid_clock.cc:648] HybridClock initialized: now 1790297570919579 us; error 31 us; skew 500 ppm
Sep 25 00:52:50 dist-test-slave-05m0 krb5kdc[23140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:50.922449 23153 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:50.923648 23153 webserver.cc:460] Webserver started at http://127.0.249.126:36455/ using document root <none> and password file <none>
I20260925 00:52:50.924185 23153 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:50.924244 23153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:50.924480 23153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:50.926506 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "0c41cb97cb324cb8af347116d943f777"
format_stamp: "Formatted at 2026-09-25 00:52:50 on dist-test-slave-05m0"
server_key: "5bf0ba855276de18f5fcdd75dc662ad4"
server_key_iv: "f2502daedea0992f8e31e08fbe7edc72"
server_key_version: "encryptionkey@0"
I20260925 00:52:50.927040 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "0c41cb97cb324cb8af347116d943f777"
format_stamp: "Formatted at 2026-09-25 00:52:50 on dist-test-slave-05m0"
server_key: "5bf0ba855276de18f5fcdd75dc662ad4"
server_key_iv: "f2502daedea0992f8e31e08fbe7edc72"
server_key_version: "encryptionkey@0"
I20260925 00:52:50.930713 23153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:52:50.932963 23168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:50.934278 23153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:50.934434 23153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "0c41cb97cb324cb8af347116d943f777"
format_stamp: "Formatted at 2026-09-25 00:52:50 on dist-test-slave-05m0"
server_key: "5bf0ba855276de18f5fcdd75dc662ad4"
server_key_iv: "f2502daedea0992f8e31e08fbe7edc72"
server_key_version: "encryptionkey@0"
I20260925 00:52:50.934553 23153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:50.938927 23153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:50.941733 23153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:50.941923 23153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:50.948545 23153 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:46645
I20260925 00:52:50.948558 23220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:46645 every 8 connection(s)
I20260925 00:52:50.949458 23153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:50.952445 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 23153
I20260925 00:52:50.952567 23221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:50.952625 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:50.952979 997 external_mini_cluster.cc:1428] Setting key 71da90af785cf432dfd6f75ff64c00fe
I20260925 00:52:50.958896 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:50.954377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41401 (local address 127.0.249.126:46645)
0925 00:52:50.954810 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:50.954845 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:50.954898 (+ 53us) server_negotiation.cc:407] Connection header received
0925 00:52:50.955710 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:50.955734 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:50.956108 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:50.956508 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:50.957252 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:50.957259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:50.957275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:50.957289 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:50.957354 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:50.957385 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:50.957627 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:50.957713 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 00:52:50.957876 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":103,"threads_started":1}
I20260925 00:52:50.960497 23221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777: Bootstrap starting.
I20260925 00:52:50.961666 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:50.963008 23221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:50.963752 23221 log.cc:824] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:50.965298 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:50.963747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54043 (local address 127.0.249.126:46645)
0925 00:52:50.963947 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:50.963951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:50.964012 (+ 61us) server_negotiation.cc:407] Connection header received
0925 00:52:50.964132 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:50.964134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:50.964159 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:50.964247 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:50.964936 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:50.964940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:50.964942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 00:52:50.964953 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:50.964978 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 00:52:50.964994 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:50.965080 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:50.965155 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 00:52:50.965191 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260925 00:52:50.965711 23221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777: No bootstrap required, opened a new log
I20260925 00:52:50.969058 23221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41cb97cb324cb8af347116d943f777" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46645 } }
I20260925 00:52:50.969300 23221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 00:52:50.969388 23221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c41cb97cb324cb8af347116d943f777, State: Initialized, Role: FOLLOWER
I20260925 00:52:50.969816 23221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41cb97cb324cb8af347116d943f777" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46645 } }
I20260925 00:52:50.969961 23221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 00:52:50.970042 23221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 00:52:50.970160 23221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:50.971014 23221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41cb97cb324cb8af347116d943f777" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46645 } }
I20260925 00:52:50.971356 23221 leader_election.cc:302] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c41cb97cb324cb8af347116d943f777; no voters:
I20260925 00:52:50.971674 23221 leader_election.cc:288] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 00:52:50.971774 23231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:50.972033 23231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [term 1 LEADER]: Becoming Leader. State: Replica: 0c41cb97cb324cb8af347116d943f777, State: Running, Role: LEADER
I20260925 00:52:50.972349 23231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41cb97cb324cb8af347116d943f777" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46645 } }
I20260925 00:52:50.972877 23221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:50.973985 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c41cb97cb324cb8af347116d943f777. Latest consensus state: current_term: 1 leader_uuid: "0c41cb97cb324cb8af347116d943f777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41cb97cb324cb8af347116d943f777" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46645 } } }
I20260925 00:52:50.974113 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:50.974388 23232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c41cb97cb324cb8af347116d943f777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41cb97cb324cb8af347116d943f777" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 46645 } } }
I20260925 00:52:50.974463 23232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:50.974532 23240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:50.977634 23240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:50.983173 23240 catalog_manager.cc:1347] Generated new cluster ID: 2ccd581d7b8c407499ecf65c6bb1c6c5
I20260925 00:52:50.983243 23240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:50.991266 23240 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:50.991520 23240 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:50.995635 23240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c41cb97cb324cb8af347116d943f777: Generated new TSK 0
I20260925 00:52:50.996295 23240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 00:52:51.006322 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 23153
2026-09-25T00:52:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1391 ms)
[----------] 18 tests from AuthTokenIssuingTest (25524 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:51 dist-test-slave-05m0 krb5kdc[23252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:51 dist-test-slave-05m0 krb5kdc[23252](info): set up 2 sockets
Sep 25 00:52:51 dist-test-slave-05m0 krb5kdc[23252](info): commencing operation
krb5kdc: starting...
W20260925 00:52:52.129882 997 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:52.250615 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:34075
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:34075,127.0.249.125:42907,127.0.249.124:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:52.358952 23265 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:52:52.359241 23265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:52.359308 23265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:52.359391 23265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:52.363823 23265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:52.363893 23265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:52.363914 23265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:52.363976 23265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:52.364017 23265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:52.369375 23265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:34075,127.0.249.125:42907,127.0.249.124:38515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:34075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:52.370481 23265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:52.371527 23265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:52.380368 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:52.380336 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:52.380445 23265 server_base.cc:1034] running on GCE node
W20260925 00:52:52.380332 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:52.381325 23265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:52.382513 23265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:52.383737 23265 hybrid_clock.cc:648] HybridClock initialized: now 1790297572383725 us; error 30 us; skew 500 ppm
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:52.386763 23265 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:52.387528 23265 webserver.cc:460] Webserver started at http://127.0.249.126:43421/ using document root <none> and password file <none>
I20260925 00:52:52.388019 23265 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:52.388073 23265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:52.388230 23265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:52.390218 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "6e7c7b026f5141c49220a0bd61806e23"
format_stamp: "Formatted at 2026-09-25 00:52:52 on dist-test-slave-05m0"
server_key: "ef017ae36bc25462a10cb27d21352dfc"
server_key_iv: "88755b0c0f5970fd7d4f641adf8c0aad"
server_key_version: "encryptionkey@0"
I20260925 00:52:52.390710 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "6e7c7b026f5141c49220a0bd61806e23"
format_stamp: "Formatted at 2026-09-25 00:52:52 on dist-test-slave-05m0"
server_key: "ef017ae36bc25462a10cb27d21352dfc"
server_key_iv: "88755b0c0f5970fd7d4f641adf8c0aad"
server_key_version: "encryptionkey@0"
I20260925 00:52:52.394193 23265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:52:52.396443 23280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:52.397702 23265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:52.397807 23265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "6e7c7b026f5141c49220a0bd61806e23"
format_stamp: "Formatted at 2026-09-25 00:52:52 on dist-test-slave-05m0"
server_key: "ef017ae36bc25462a10cb27d21352dfc"
server_key_iv: "88755b0c0f5970fd7d4f641adf8c0aad"
server_key_version: "encryptionkey@0"
I20260925 00:52:52.397887 23265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:52.410894 23265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:52.413663 23265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:52.413820 23265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:52.420367 23265 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:34075
I20260925 00:52:52.420641 23332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:34075 every 8 connection(s)
I20260925 00:52:52.421298 23265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:52.424005 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:52.426443 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 23265
I20260925 00:52:52.426555 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:52.426839 997 external_mini_cluster.cc:1428] Setting key c52b50c941e87e488b2698570b1f07d6
I20260925 00:52:52.427840 23333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.249.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.125@KRBTEST.COM" created.
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:52.446305 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.430383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.0.249.126:34075)
0925 00:52:52.430972 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.430986 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.431032 (+ 46us) server_negotiation.cc:407] Connection header received
0925 00:52:52.432055 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.432065 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.432315 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.432538 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.433485 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.434949 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.436587 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.437219 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.437453 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.440682 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.440691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.440708 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.440738 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.443023 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.443549 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.443558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.443562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.443715 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.444095 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.444098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.444099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.444500 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.444606 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.445119 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.445385 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":453,"spinlock_wait_cycles":30592,"thread_start_us":166,"threads_started":1}
I20260925 00:52:52.446310 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.430382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:34075 (local address 127.0.0.1:40402)
0925 00:52:52.430849 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.430901 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:52.431737 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:52.432538 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:52.432543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:52.432878 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:52.433260 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.433280 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.435063 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.435066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.436456 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.436470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.437459 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.437466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.437630 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.438354 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:52.438381 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:52.440428 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:52.443148 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.443160 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.443171 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.443429 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.443830 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.443836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.443841 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.443991 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.444624 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:52.444634 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:52.444842 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 00:52:52.445062 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 00:52:52.445484 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":64,"thread_start_us":140,"threads_started":1}
I20260925 00:52:52.448921 23333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:52.449797 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.449447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42907 (local address 127.0.0.1:38848)
0925 00:52:52.449576 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.449681 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 00:52:52.450008 23282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
I20260925 00:52:52.450848 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.450655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42907 (local address 127.0.0.1:38856)
0925 00:52:52.450752 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.450803 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 00:52:52.450994 23333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.125:42907: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab.
Entry for principal kudu/127.0.249.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab.
I20260925 00:52:52.456409 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.125:42907
--webserver_interface=127.0.249.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab
--principal=kudu/127.0.249.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:34075,127.0.249.125:42907,127.0.249.124:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:52:52.500260 23333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } attempt: 1
I20260925 00:52:52.501132 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.500839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42907 (local address 127.0.0.1:38862)
0925 00:52:52.500980 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.501050 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 00:52:52.501926 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.501704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42907 (local address 127.0.0.1:38866)
0925 00:52:52.501824 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.501870 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 00:52:52.502128 23333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.125:42907: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
W20260925 00:52:52.565950 23340 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:52:52.566255 23340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:52.566279 23340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:52.566318 23340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 00:52:52.570319 23333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } attempt: 2
W20260925 00:52:52.570432 23340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:52.570482 23340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:52.570508 23340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:52.570528 23340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:52.570549 23340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:52.571192 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.570867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42907 (local address 127.0.0.1:38878)
0925 00:52:52.571043 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.571113 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 00:52:52.572019 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.571819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42907 (local address 127.0.0.1:38894)
0925 00:52:52.571921 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.571965 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 00:52:52.572180 23333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.125:42907: Network error: Client connection negotiation failed: client connection to 127.0.249.125:42907: connect: Connection refused (error 111)
I20260925 00:52:52.576498 23340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:34075,127.0.249.125:42907,127.0.249.124:38515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab
--principal=kudu/127.0.249.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.125:42907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.249.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:52.577711 23340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:52.578774 23340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:52.587419 23345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:52.587445 23348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:52.587419 23346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:52.587659 23340 server_base.cc:1034] running on GCE node
I20260925 00:52:52.588301 23340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:52.589375 23340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:52.590595 23340 hybrid_clock.cc:648] HybridClock initialized: now 1790297572590580 us; error 26 us; skew 500 ppm
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:52.593359 23340 init.cc:377] Logged in from keytab as kudu/127.0.249.125@KRBTEST.COM (short username kudu)
I20260925 00:52:52.594259 23340 webserver.cc:460] Webserver started at http://127.0.249.125:33285/ using document root <none> and password file <none>
I20260925 00:52:52.594810 23340 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:52.594868 23340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:52.595106 23340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:52.597106 23340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/data/instance:
uuid: "206b38443c504220b565329d61938b92"
format_stamp: "Formatted at 2026-09-25 00:52:52 on dist-test-slave-05m0"
server_key: "8132359877fbd3004d4993150aa684c9"
server_key_iv: "8fd82e3d286b08c229f3acaad6ff7941"
server_key_version: "encryptionkey@0"
I20260925 00:52:52.597728 23340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/wal/instance:
uuid: "206b38443c504220b565329d61938b92"
format_stamp: "Formatted at 2026-09-25 00:52:52 on dist-test-slave-05m0"
server_key: "8132359877fbd3004d4993150aa684c9"
server_key_iv: "8fd82e3d286b08c229f3acaad6ff7941"
server_key_version: "encryptionkey@0"
I20260925 00:52:52.601702 23340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 00:52:52.604169 23355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:52.605469 23340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 00:52:52.605599 23340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/wal
uuid: "206b38443c504220b565329d61938b92"
format_stamp: "Formatted at 2026-09-25 00:52:52 on dist-test-slave-05m0"
server_key: "8132359877fbd3004d4993150aa684c9"
server_key_iv: "8fd82e3d286b08c229f3acaad6ff7941"
server_key_version: "encryptionkey@0"
I20260925 00:52:52.605717 23340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:52.624912 23340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:52.628438 23340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:52.628650 23340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:52.635614 23340 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.125:42907
I20260925 00:52:52.635910 23407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.125:42907 every 8 connection(s)
I20260925 00:52:52.636593 23340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
I20260925 00:52:52.639276 23408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:52.642753 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 23340
I20260925 00:52:52.642865 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-1/wal/instance
I20260925 00:52:52.643169 997 external_mini_cluster.cc:1428] Setting key ab181fb25dd1f92a6763b93f208caee3
I20260925 00:52:52.643122 23408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.249.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.124@KRBTEST.COM" created.
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:52.659008 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.645459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.0.249.126:34075)
0925 00:52:52.645591 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.645595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.646653 (+ 1058us) server_negotiation.cc:407] Connection header received
0925 00:52:52.647553 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.647559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.647633 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.647745 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.648927 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.649858 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.650980 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.651387 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.651502 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.655361 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.655364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.655367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.655390 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.656866 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.657424 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.657427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.657429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.657526 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.657872 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.657877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.657879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.658068 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.658179 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.658707 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.658832 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 00:52:52.659835 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.645934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:34075 (local address 127.0.0.1:40404)
0925 00:52:52.646454 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.646523 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:52.647349 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:52.647896 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:52.647902 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:52.648215 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:52.648745 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.648773 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.649986 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.649988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.650846 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.650855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.651527 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.651530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.651712 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.652246 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:52.652260 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:52.655170 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:52.656996 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.657002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.657017 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.657287 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.657622 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.657626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.657628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.657774 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.658187 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:52.658196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:52.658444 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 00:52:52.658644 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 00:52:52.658963 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":159,"threads_started":1}
I20260925 00:52:52.661514 23408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:52.663338 23333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } attempt: 3
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab.
Entry for principal kudu/127.0.249.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab.
I20260925 00:52:52.670492 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.124:38515
--webserver_interface=127.0.249.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab
--principal=kudu/127.0.249.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:34075,127.0.249.125:42907,127.0.249.124:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:52:52.672952 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.662007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42907 (local address 127.0.0.1:38898)
0925 00:52:52.662127 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.662141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:52.662214 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:52.662760 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:52.662763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:52.662895 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:52.663019 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.663026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.664258 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.664261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.665098 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.665107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.665975 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.665978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.666107 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.666587 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:52.666598 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:52.668116 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:52.671020 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.671027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.671033 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.671333 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.671636 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.671643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.671648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.671762 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.672200 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:52.672205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:52.672334 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 00:52:52.672475 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 00:52:52.672697 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":48}
I20260925 00:52:52.673080 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.662117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.0.249.125:42907)
0925 00:52:52.662439 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.662448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.662467 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:52:52.662501 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.662504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.662645 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.662755 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.663112 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.664141 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.665224 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.665810 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.665965 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.668304 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.668314 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.668328 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.668360 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.670909 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.671442 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.671446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.671448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.671527 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.671858 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.671860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.671861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.672087 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.672191 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.672549 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.672814 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":204,"thread_start_us":133,"threads_started":1}
I20260925 00:52:52.675123 23408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:52.675506 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.663723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42907 (local address 127.0.0.1:38910)
0925 00:52:52.663857 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.663870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:52.663941 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:52.664413 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:52.664416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:52.664585 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:52.664701 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.664707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.665511 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.665515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.666267 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.666276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.666828 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.666832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.666940 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.667359 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:52.667379 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:52.668637 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:52.673124 (+ 4487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.673127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.673128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.673360 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.674369 (+ 1009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.674372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.674374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.674462 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.674944 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:52.674949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:52.675063 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 00:52:52.675190 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 00:52:52.675342 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 00:52:52.675662 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.663893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.0.249.125:42907)
0925 00:52:52.664153 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.664159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.664170 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:52:52.664195 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.664197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.664225 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.664347 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.664812 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.665372 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.666391 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.666690 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.666824 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.668797 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.668803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.668807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.668838 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.672994 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.674163 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.674166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.674167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.674261 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.674581 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.674587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.674589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.674817 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.674942 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.675265 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.675394 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":109,"threads_started":1}
I20260925 00:52:52.676035 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.675697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53508)
0925 00:52:52.675898 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.675973 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260925 00:52:52.676250 23358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
I20260925 00:52:52.676276 23333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:52.677197 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.676899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53514)
0925 00:52:52.677030 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.677090 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 00:52:52.678056 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.677628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53526)
0925 00:52:52.677900 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.677958 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":101,"threads_started":1}
W20260925 00:52:52.678242 23408 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:38515: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
I20260925 00:52:52.678328 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.677973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53528)
0925 00:52:52.678179 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.678243 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260925 00:52:52.678485 23333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:38515: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
I20260925 00:52:52.709646 23333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } attempt: 1
I20260925 00:52:52.710516 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.710160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53534)
0925 00:52:52.710351 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.710423 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 00:52:52.711220 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.711002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53538)
0925 00:52:52.711114 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.711165 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 00:52:52.711377 23333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:38515: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
I20260925 00:52:52.727515 23408 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } attempt: 1
I20260925 00:52:52.728341 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.728065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53540)
0925 00:52:52.728209 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.728270 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 00:52:52.729012 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.728806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53552)
0925 00:52:52.728909 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.728957 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 00:52:52.729135 23408 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:38515: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
W20260925 00:52:52.782814 23415 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:52:52.783139 23415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:52.783215 23415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:52.783291 23415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 00:52:52.786543 23333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } attempt: 2
I20260925 00:52:52.787360 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.787029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53564)
0925 00:52:52.787196 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.787270 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 00:52:52.787564 23415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:52.787621 23415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:52.787648 23415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:52.787674 23415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:52.787699 23415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:52.788136 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.787953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53568)
0925 00:52:52.788037 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.788082 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 00:52:52.788350 23333 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:38515: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
I20260925 00:52:52.793081 23415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:34075,127.0.249.125:42907,127.0.249.124:38515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab
--principal=kudu/127.0.249.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.124:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.249.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:52.794186 23415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:52.795219 23415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:52:52.797367 23408 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } attempt: 2
I20260925 00:52:52.798120 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.797869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53570)
0925 00:52:52.797997 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.798056 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 00:52:52.798823 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.798615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53580)
0925 00:52:52.798719 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.798763 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 00:52:52.798949 23408 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:38515: Network error: Client connection negotiation failed: client connection to 127.0.249.124:38515: connect: Connection refused (error 111)
W20260925 00:52:52.803916 23423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:52.803953 23425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:52.804049 23415 server_base.cc:1034] running on GCE node
W20260925 00:52:52.804204 23422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:52.804723 23415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:52.805838 23415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:52.807026 23415 hybrid_clock.cc:648] HybridClock initialized: now 1790297572807010 us; error 25 us; skew 500 ppm
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:52.809772 23415 init.cc:377] Logged in from keytab as kudu/127.0.249.124@KRBTEST.COM (short username kudu)
I20260925 00:52:52.810674 23415 webserver.cc:460] Webserver started at http://127.0.249.124:45433/ using document root <none> and password file <none>
I20260925 00:52:52.811223 23415 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:52.811316 23415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:52.811523 23415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:52.813719 23415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/data/instance:
uuid: "642df75b6e3c4e5d991d9ba84ed98c92"
format_stamp: "Formatted at 2026-09-25 00:52:52 on dist-test-slave-05m0"
server_key: "62e7404fa1dc39a511cb8af8f4b2a4b6"
server_key_iv: "49111b8f05a8060b07dfd299306d05ad"
server_key_version: "encryptionkey@0"
I20260925 00:52:52.814281 23415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/wal/instance:
uuid: "642df75b6e3c4e5d991d9ba84ed98c92"
format_stamp: "Formatted at 2026-09-25 00:52:52 on dist-test-slave-05m0"
server_key: "62e7404fa1dc39a511cb8af8f4b2a4b6"
server_key_iv: "49111b8f05a8060b07dfd299306d05ad"
server_key_version: "encryptionkey@0"
I20260925 00:52:52.817836 23415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 00:52:52.820128 23432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:52.821256 23415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:52.821434 23415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/wal
uuid: "642df75b6e3c4e5d991d9ba84ed98c92"
format_stamp: "Formatted at 2026-09-25 00:52:52 on dist-test-slave-05m0"
server_key: "62e7404fa1dc39a511cb8af8f4b2a4b6"
server_key_iv: "49111b8f05a8060b07dfd299306d05ad"
server_key_version: "encryptionkey@0"
I20260925 00:52:52.821556 23415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:52.840917 23415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:52.843982 23415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:52.844200 23415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:52.851399 23415 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.124:38515
I20260925 00:52:52.851670 23484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.124:38515 every 8 connection(s)
I20260925 00:52:52.852382 23415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
I20260925 00:52:52.855041 23485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:52.858450 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 23415
I20260925 00:52:52.858564 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790297292685301-997-0/minicluster-data/master-2/wal/instance
I20260925 00:52:52.858852 997 external_mini_cluster.cc:1428] Setting key 48cd6a658bf6138f3be1a0d2de988e9c
I20260925 00:52:52.858667 23485 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
I20260925 00:52:52.872318 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.860704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.0.249.126:34075)
0925 00:52:52.860851 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.860855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.861571 (+ 716us) server_negotiation.cc:407] Connection header received
0925 00:52:52.861849 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.861854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.861903 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.861979 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.863024 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.863792 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.865113 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.865466 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.865562 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.868284 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.868287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.868289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.868311 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.870061 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.870587 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.870593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.870595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.870697 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.871094 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.871096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.871097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.871359 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.871448 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.872011 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.872153 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:52:52.873185 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.860624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:34075 (local address 127.0.0.1:40410)
0925 00:52:52.861384 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.861488 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:52.861689 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:52.862054 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:52.862061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:52.862536 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:52.862898 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.862911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.863894 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.863896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.864995 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.865006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.865608 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.865611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.865755 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.866309 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:52.866352 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:52.868122 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:52.870166 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.870173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.870180 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.870481 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.870805 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.870808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.870811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.870996 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.871465 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:52.871471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:52.871745 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 00:52:52.871942 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 00:52:52.872255 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":126,"threads_started":1}
I20260925 00:52:52.873193 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.860152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.0.249.124:38515)
0925 00:52:52.860602 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.860621 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.860659 (+ 38us) server_negotiation.cc:407] Connection header received
0925 00:52:52.861359 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.861375 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.861709 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.861911 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.862867 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.864069 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.865240 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.865776 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.865851 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.868778 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.868786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.868798 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.868822 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.870636 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.871211 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.871217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.871218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.871298 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.871595 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.871597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.871599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.871818 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.872001 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.872233 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.872368 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
I20260925 00:52:52.874428 23485 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:52.884145 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.874921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:42907 (local address 127.0.0.1:38920)
0925 00:52:52.875064 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.875076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:52.875149 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:52.875642 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:52.875646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:52.875825 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:52.875920 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.875925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.876958 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.876961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.877851 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.877862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.878468 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.878472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.878569 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.879069 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:52.879087 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:52.880573 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:52.882575 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.882580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.882582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.882817 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.883123 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.883126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.883128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.883217 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.883642 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:52.883645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:52.883730 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 00:52:52.883848 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 00:52:52.883980 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 00:52:52.884248 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.875043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.0.249.125:42907)
0925 00:52:52.875167 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.875176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.875191 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:52:52.875249 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.875252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.875305 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.875391 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.876051 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.876852 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.877985 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.878361 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.878457 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.880722 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.880727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.880730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.880764 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.882465 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.882917 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.882921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.882923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.883010 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.883313 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.883318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.883319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.883541 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.883624 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.883917 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.884054 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 00:52:52.884344 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.875369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.0.249.126:34075)
0925 00:52:52.875540 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.875544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.875557 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:52:52.875601 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.875605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.875642 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.875729 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.876251 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.876949 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.877994 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.878338 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.878449 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.881237 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.881240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.881243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.881277 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.882682 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.883139 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.883143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.883144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.883217 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.883486 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.883488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.883489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.883686 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.883923 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.884053 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.884151 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 00:52:52.885605 23485 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:52.887502 23333 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } attempt: 3
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
I20260925 00:52:52.890095 23408 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } attempt: 3
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
I20260925 00:52:52.893994 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.886085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53612)
0925 00:52:52.886165 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.886179 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:52.886246 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:52.886470 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:52.886471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:52.886585 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:52.886647 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.886651 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.887334 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.887340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.888007 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.888020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.888492 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.888496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.888611 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.888989 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:52.889002 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:52.890328 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:52.892343 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.892348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.892353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.892598 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.892902 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.892905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.892908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.892998 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.893482 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:52.893488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:52.893578 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 00:52:52.893681 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 00:52:52.893811 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":32}
I20260925 00:52:52.894562 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.886136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.0.249.124:38515)
0925 00:52:52.886273 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.886278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.886292 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:52:52.886332 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.886335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.886379 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.886466 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.886751 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.887234 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.888132 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.888390 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.888496 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.890451 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.890454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.890457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.890478 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.892216 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.892709 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.892715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.892718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.892794 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.893099 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.893104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.893109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.893313 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.893443 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.894290 (+ 847us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.894402 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":59}
I20260925 00:52:52.895818 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.885488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.0.249.125:42907)
0925 00:52:52.885600 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.885604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.885622 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:52:52.885814 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.885818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.885858 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.885948 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.886620 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.887089 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.888265 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.888570 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.889244 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.891655 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.891659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.891661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.891688 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.893748 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.894235 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.894238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.894240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.894318 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.894622 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.894625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.894627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.894811 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.895028 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.895150 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.895255 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 00:52:52 dist-test-slave-05m0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
I20260925 00:52:52.898779 23485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92: Bootstrap starting.
I20260925 00:52:52.899729 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.887876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53624)
0925 00:52:52.887974 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.887988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:52.888054 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:52.888686 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:52.888689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:52.888859 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:52.888980 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.888987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.890634 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.890640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.891497 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.891509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.892009 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.892012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.892098 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.892672 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:52.892692 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:52.895831 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:52.898024 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.898028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.898033 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.898288 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.898605 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.898611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.898616 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.898709 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.899249 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:52.899253 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:52.899355 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 00:52:52.899460 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 00:52:52.899586 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 00:52:52.900166 23490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.887932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.0.249.124:38515)
0925 00:52:52.888438 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.888446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.888457 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:52:52.888487 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.888490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.888527 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.888665 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.889114 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.890486 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.891647 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.891894 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.892006 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.896114 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.896117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.896120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.896144 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.897916 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.898400 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.898404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.898406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.898496 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.898846 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.898850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.898852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.899105 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.899199 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.899943 (+ 744us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.900043 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":112,"threads_started":1}
I20260925 00:52:52.901377 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.890429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:38515 (local address 127.0.0.1:53626)
0925 00:52:52.890585 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:52.890601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:52.890690 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:52.892425 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:52.892428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:52.892720 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:52.892865 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.892872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.894008 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.894012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.894953 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:52.894967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.895562 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.895566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:52.895659 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.896138 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:52.896153 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:52.897547 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:52.899629 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.899632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.899635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.899884 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.900442 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:52.900445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:52.900447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:52.900530 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:52.900933 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:52.900936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:52.901023 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 00:52:52.901101 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 00:52:52.901238 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 00:52:52.901587 23485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:52.901916 23491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.890888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.0.249.124:38515)
0925 00:52:52.892197 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.892207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.892221 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:52:52.892256 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.892262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.892303 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.892407 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.892999 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.893882 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.895114 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.895444 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.895559 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.897685 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.897689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.897691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.897714 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.899511 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.900247 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.900250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.900252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.900334 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.900638 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.900640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.900642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.900821 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.901498 (+ 677us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.901577 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.901677 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1242,"thread_start_us":114,"threads_started":1}
I20260925 00:52:52.903082 23333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23: Bootstrap starting.
I20260925 00:52:52.905078 23408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92: Bootstrap starting.
I20260925 00:52:52.905375 23333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:52.906102 23333 log.cc:824] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:52.907290 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:52.907476 23408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:52.908198 23333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23: No bootstrap required, opened a new log
I20260925 00:52:52.908309 23408 log.cc:824] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:52.909221 23485 log.cc:824] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:52.908563 23307 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6e7c7b026f5141c49220a0bd61806e23"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40412
I20260925 00:52:52.910835 23408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92: No bootstrap required, opened a new log
I20260925 00:52:52.911145 23485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92: No bootstrap required, opened a new log
I20260925 00:52:52.911902 23333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } }
I20260925 00:52:52.912117 23333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7c7b026f5141c49220a0bd61806e23, State: Initialized, Role: FOLLOWER
I20260925 00:52:52.912427 23333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } }
I20260925 00:52:52.912904 23495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.913141 23495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:52.913738 23333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:52.914861 23408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } }
I20260925 00:52:52.915077 23408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206b38443c504220b565329d61938b92, State: Initialized, Role: FOLLOWER
W20260925 00:52:52.915146 23286 tablet.cc:2369] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:52.915378 23408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } }
I20260925 00:52:52.915618 23307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 00:52:52.915729 23307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:52.915674 23485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } }
I20260925 00:52:52.915820 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.915891 23485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642df75b6e3c4e5d991d9ba84ed98c92, State: Initialized, Role: FOLLOWER
I20260925 00:52:52.915942 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:52.916167 23485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } }
I20260925 00:52:52.916572 23408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:52.916610 23502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.916724 23502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:52.916952 23307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } }
I20260925 00:52:52.917343 23485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:52.918367 23382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e7c7b026f5141c49220a0bd61806e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "206b38443c504220b565329d61938b92"
I20260925 00:52:52.918645 23382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:52.918835 23459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e7c7b026f5141c49220a0bd61806e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "642df75b6e3c4e5d991d9ba84ed98c92"
I20260925 00:52:52.919056 23459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:52.919749 23382 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7c7b026f5141c49220a0bd61806e23 in term 1.
I20260925 00:52:52.920243 23459 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7c7b026f5141c49220a0bd61806e23 in term 1.
I20260925 00:52:52.920202 23282 leader_election.cc:302] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206b38443c504220b565329d61938b92, 6e7c7b026f5141c49220a0bd61806e23; no voters:
I20260925 00:52:52.920480 23495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:52.920562 23307 leader_election.cc:288] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [CANDIDATE]: Term 1 election: Requested vote from peers 206b38443c504220b565329d61938b92 (127.0.249.125:42907), 642df75b6e3c4e5d991d9ba84ed98c92 (127.0.249.124:38515)
I20260925 00:52:52.920679 23495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7c7b026f5141c49220a0bd61806e23, State: Running, Role: LEADER
I20260925 00:52:52.920976 23495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } }
I20260925 00:52:52.923635 23524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e7c7b026f5141c49220a0bd61806e23. Latest consensus state: current_term: 1 leader_uuid: "6e7c7b026f5141c49220a0bd61806e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.923760 23524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:52.924446 23533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260925 00:52:52.926465 23523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:52.926589 23523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:52.927398 23533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 00:52:52.928339 23530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:52.928496 23530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:52.922196 23340 master_runner.cc:186] Error getting master registration for 127.0.249.126:34075: OK, instance_id { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" instance_seqno: 1790297572418646 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 00:52:52.928983 23340 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:52:52.932920 23459 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [term 1 FOLLOWER]: Refusing update from remote peer 6e7c7b026f5141c49220a0bd61806e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:52.932973 23382 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [term 1 FOLLOWER]: Refusing update from remote peer 6e7c7b026f5141c49220a0bd61806e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:52.933647 23524 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:52.933815 23495 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:52.937120 23502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e7c7b026f5141c49220a0bd61806e23. Latest consensus state: current_term: 1 leader_uuid: "6e7c7b026f5141c49220a0bd61806e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.937224 23502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:52.937163 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e7c7b026f5141c49220a0bd61806e23. Latest consensus state: current_term: 1 leader_uuid: "6e7c7b026f5141c49220a0bd61806e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.937300 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:52.941570 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.929506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.0.249.126:34075)
0925 00:52:52.929604 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.929608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.931824 (+ 2216us) server_negotiation.cc:407] Connection header received
0925 00:52:52.932574 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.932577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.932627 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.932689 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.933727 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.934428 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.935545 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.935871 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.935947 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.937889 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.937893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.937896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.937920 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.939659 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.940118 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.940121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.940123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.940215 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.940512 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.940516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.940518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.940716 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.940830 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.941199 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.941333 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 00:52:52.946529 23495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e7c7b026f5141c49220a0bd61806e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.946651 23495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:52.946805 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.930432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.0.249.125:42907)
0925 00:52:52.930529 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.930534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.932157 (+ 1623us) server_negotiation.cc:407] Connection header received
0925 00:52:52.932330 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.932335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.932380 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.932432 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.938477 (+ 6045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.939335 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.940458 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.940794 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.940865 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.942831 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.942835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.942838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.942869 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.944876 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.945410 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.945414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.945415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.945502 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.945821 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.945827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.945832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.946055 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.946153 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.946515 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.946633 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 00:52:52.947171 23491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:52.931349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.0.249.124:38515)
0925 00:52:52.931453 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:52.931457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:52.938355 (+ 6898us) server_negotiation.cc:407] Connection header received
0925 00:52:52.938534 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:52.938537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:52.938581 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:52.938628 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:52.939438 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.940241 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.941489 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:52.941854 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:52.941931 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:52.943637 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:52.943644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:52.943652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:52.943682 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:52.945378 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.945848 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.945852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.945853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.945928 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:52.946243 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:52.946247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:52.946248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:52.946466 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:52.946557 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:52.946896 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 00:52:52.947015 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 00:52:52.948283 23502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e7c7b026f5141c49220a0bd61806e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.948390 23502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:52.948820 23541 mvcc.cc:205] Tried to move back new op lower bound from 7333058858727120896 to 7333058858687746048. Current Snapshot: MvccSnapshot[applied={T|T < 7333058858727120896}]
I20260925 00:52:52.949216 23545 mvcc.cc:205] Tried to move back new op lower bound from 7333058858727120896 to 7333058858687746048. Current Snapshot: MvccSnapshot[applied={T|T < 7333058858727120896}]
I20260925 00:52:52.952906 23524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e7c7b026f5141c49220a0bd61806e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.953037 23524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23 [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:52.953790 23533 catalog_manager.cc:1347] Generated new cluster ID: 85aa18166ce74651b6a8b93bc5505dd4
I20260925 00:52:52.953850 23533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:52.954037 23546 mvcc.cc:205] Tried to move back new op lower bound from 7333058858727120896 to 7333058858687746048. Current Snapshot: MvccSnapshot[applied={T|T < 7333058858727120896}]
I20260925 00:52:52.954222 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e7c7b026f5141c49220a0bd61806e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7c7b026f5141c49220a0bd61806e23" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 34075 } } peers { permanent_uuid: "206b38443c504220b565329d61938b92" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 42907 } } peers { permanent_uuid: "642df75b6e3c4e5d991d9ba84ed98c92" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 38515 } } }
I20260925 00:52:52.954322 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:52.985612 23533 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:52.986912 23533 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:53.007898 23533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e7c7b026f5141c49220a0bd61806e23: Generated new TSK 0
I20260925 00:52:53.008935 23533 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:52:53.052313 23491 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.049184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.0.249.124:38515)
0925 00:52:53.049313 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.049318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.051219 (+ 1901us) server_negotiation.cc:407] Connection header received
0925 00:52:53.051425 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.051429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.051480 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.051533 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.052174 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53656: BlockingRecv error: recv got EOF from 127.0.0.1:53656 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 00:52:53.054225 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.050001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.0.249.126:34075)
0925 00:52:53.050111 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.050115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.053038 (+ 2923us) server_negotiation.cc:407] Connection header received
0925 00:52:53.053428 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.053431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.053479 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.053529 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.054112 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40434: BlockingRecv error: recv got EOF from 127.0.0.1:40434 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 00:52:53.055307 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.050611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.0.249.125:42907)
0925 00:52:53.050723 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.050727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.053231 (+ 2504us) server_negotiation.cc:407] Connection header received
0925 00:52:53.054578 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.054582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.054633 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.054684 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.055195 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38948: BlockingRecv error: recv got EOF from 127.0.0.1:38948 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 00:52:53.060871 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.059384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.0.249.125:42907)
0925 00:52:53.059481 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.059485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.059948 (+ 463us) server_negotiation.cc:407] Connection header received
0925 00:52:53.060164 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.060167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.060242 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.060293 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.060761 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38952: BlockingRecv error: recv got EOF from 127.0.0.1:38952 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 00:52:53.067921 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.066296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.0.249.125:42907)
0925 00:52:53.066401 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.066405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.067000 (+ 595us) server_negotiation.cc:407] Connection header received
0925 00:52:53.067207 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.067211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.067260 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.067314 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.067819 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38964: BlockingRecv error: recv got EOF from 127.0.0.1:38964 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 00:52:53.073920 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.072314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.0.249.125:42907)
0925 00:52:53.072414 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.072419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.072909 (+ 490us) server_negotiation.cc:407] Connection header received
0925 00:52:53.073134 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.073138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.073188 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.073242 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.073814 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38980: BlockingRecv error: recv got EOF from 127.0.0.1:38980 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 00:52:53.081964 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.080261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.0.249.125:42907)
0925 00:52:53.080388 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.080392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.080907 (+ 515us) server_negotiation.cc:407] Connection header received
0925 00:52:53.081116 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.081120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.081169 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.081223 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.081859 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38984: BlockingRecv error: recv got EOF from 127.0.0.1:38984 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 00:52:53.094478 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.092598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.0.249.125:42907)
0925 00:52:53.092726 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.092732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.093348 (+ 616us) server_negotiation.cc:407] Connection header received
0925 00:52:53.093626 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.093630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.093685 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.093744 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.094363 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38990: BlockingRecv error: recv got EOF from 127.0.0.1:38990 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 00:52:53.115727 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.113522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.0.249.125:42907)
0925 00:52:53.113661 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.113667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.114460 (+ 793us) server_negotiation.cc:407] Connection header received
0925 00:52:53.114711 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.114715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.114777 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.114840 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.115604 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38994: BlockingRecv error: recv got EOF from 127.0.0.1:38994 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 00:52:53.152529 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.150783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.0.249.125:42907)
0925 00:52:53.150904 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.150909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.151497 (+ 588us) server_negotiation.cc:407] Connection header received
0925 00:52:53.151717 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.151721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.151780 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.151839 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.152407 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39002: BlockingRecv error: recv got EOF from 127.0.0.1:39002 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 00:52:53.220032 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.218806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.249.125:42907)
0925 00:52:53.218943 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.218949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.219131 (+ 182us) server_negotiation.cc:407] Connection header received
0925 00:52:53.219295 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.219298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.219357 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.219446 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.219917 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39012: BlockingRecv error: recv got EOF from 127.0.0.1:39012 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 00:52:53.351469 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.350159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.0.249.125:42907)
0925 00:52:53.350328 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.350334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.350525 (+ 191us) server_negotiation.cc:407] Connection header received
0925 00:52:53.350713 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.350719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.350783 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.350877 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.351358 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39014: BlockingRecv error: recv got EOF from 127.0.0.1:39014 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 00:52:53.610791 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:53.609694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.0.249.125:42907)
0925 00:52:53.609832 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:53.609836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:53.609979 (+ 143us) server_negotiation.cc:407] Connection header received
0925 00:52:53.610129 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:53.610132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:53.610179 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:53.610260 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:53.610657 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39016: BlockingRecv error: recv got EOF from 127.0.0.1:39016 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 00:52:53.927915 23523 catalog_manager.cc:1259] Loaded cluster ID: 85aa18166ce74651b6a8b93bc5505dd4
I20260925 00:52:53.927999 23523 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92: loading cluster ID for follower catalog manager: success
I20260925 00:52:53.929808 23530 catalog_manager.cc:1259] Loaded cluster ID: 85aa18166ce74651b6a8b93bc5505dd4
I20260925 00:52:53.929883 23530 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92: loading cluster ID for follower catalog manager: success
I20260925 00:52:53.930330 23523 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92: acquiring CA information for follower catalog manager: success
I20260925 00:52:53.931702 23523 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 206b38443c504220b565329d61938b92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:52:53.931951 23530 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92: acquiring CA information for follower catalog manager: success
I20260925 00:52:53.932932 23530 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 642df75b6e3c4e5d991d9ba84ed98c92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:52:54.129120 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:54.125223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.0.249.125:42907)
0925 00:52:54.125479 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:54.125483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:54.125571 (+ 88us) server_negotiation.cc:407] Connection header received
0925 00:52:54.125709 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:54.125712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:54.125834 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:54.125942 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:54.126215 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.126754 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.127626 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.127878 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.127971 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:54.128263 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:54.128699 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:54.128795 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:54.128848 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 00:52:54.128901 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
I20260925 00:52:54.135754 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:54.132205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.0.249.124:38515)
0925 00:52:54.132463 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:54.132467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:54.132481 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:52:54.132597 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:54.132600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:54.132718 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:54.132839 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:54.133037 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.133590 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.134379 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.134587 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.134661 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:54.134964 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:54.135360 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:54.135456 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:54.135508 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 00:52:54.135555 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":66,"threads_started":1}
I20260925 00:52:54.143193 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:54.139052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.0.249.125:42907)
0925 00:52:54.139213 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:54.139218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:54.139495 (+ 277us) server_negotiation.cc:407] Connection header received
0925 00:52:54.139681 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:54.139685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:54.139735 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:54.139802 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:54.140160 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.140886 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.141952 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.142288 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.142363 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:54.142704 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:54.142777 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:54.142837 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:54.142996 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 00:52:54.143061 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 00:52:54.143451 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:54.138795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.0.249.126:34075)
0925 00:52:54.139175 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:54.139179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:54.139194 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:52:54.139393 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:54.139398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:54.139571 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:54.139693 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:54.139915 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.140520 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.141646 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.141999 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.142092 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:54.142394 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:54.142779 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:54.142915 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:54.143203 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 00:52:54.143291 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":87,"threads_started":1}
I20260925 00:52:54.143765 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:54.138893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.0.249.124:38515)
0925 00:52:54.139003 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:54.139007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:54.139443 (+ 436us) server_negotiation.cc:407] Connection header received
0925 00:52:54.139531 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:54.139534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:54.139581 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:54.139654 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:54.140004 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.140785 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.141931 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:54.142215 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:54.142331 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:54.143266 (+ 935us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:54.143348 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:54.143416 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:54.143582 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 00:52:54.143643 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 00:52:54.146811 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 23265
I20260925 00:52:54.153257 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 23340
I20260925 00:52:54.159168 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 23415
2026-09-25T00:52:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3153 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 00:52:54 dist-test-slave-05m0 krb5kdc[23635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 00:52:54 dist-test-slave-05m0 krb5kdc[23635](info): set up 2 sockets
Sep 25 00:52:54 dist-test-slave-05m0 krb5kdc[23635](info): commencing operation
krb5kdc: starting...
W20260925 00:52:55.305851 997 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/test-user.keytab.
Sep 25 00:52:55 dist-test-slave-05m0 krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T00:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T00:52:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.249.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
Entry for principal kudu/127.0.249.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab.
I20260925 00:52:55.432655 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:44689
--webserver_interface=127.0.249.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:44689,127.0.249.125:37947,127.0.249.124:36619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 00:52:55.540992 23648 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:52:55.541263 23648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:55.541296 23648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:55.541333 23648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:55.545250 23648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:55.545306 23648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:55.545325 23648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:55.545342 23648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:55.545357 23648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:55.550480 23648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:44689,127.0.249.125:37947,127.0.249.124:36619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.126.keytab
--principal=kudu/127.0.249.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.126:44689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.249.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:55.551429 23648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:55.552353 23648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:55.560875 23654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:55.560982 23653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:55.560894 23656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:55.560958 23648 server_base.cc:1034] running on GCE node
I20260925 00:52:55.561782 23648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:55.562827 23648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:55.563982 23648 hybrid_clock.cc:648] HybridClock initialized: now 1790297575563947 us; error 48 us; skew 500 ppm
Sep 25 00:52:55 dist-test-slave-05m0 krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:55.566622 23648 init.cc:377] Logged in from keytab as kudu/127.0.249.126@KRBTEST.COM (short username kudu)
I20260925 00:52:55.567533 23648 webserver.cc:460] Webserver started at http://127.0.249.126:44769/ using document root <none> and password file <none>
I20260925 00:52:55.568110 23648 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:55.568200 23648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:55.568444 23648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:55.570539 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/data/instance:
uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc"
format_stamp: "Formatted at 2026-09-25 00:52:55 on dist-test-slave-05m0"
server_key: "e797ae477ddaf37a31364f3573f730fc"
server_key_iv: "63317b76b4372bd7d0e6428d68c88510"
server_key_version: "encryptionkey@0"
I20260925 00:52:55.571101 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/wal/instance:
uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc"
format_stamp: "Formatted at 2026-09-25 00:52:55 on dist-test-slave-05m0"
server_key: "e797ae477ddaf37a31364f3573f730fc"
server_key_iv: "63317b76b4372bd7d0e6428d68c88510"
server_key_version: "encryptionkey@0"
I20260925 00:52:55.574862 23648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 00:52:55.577323 23663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:55.578648 23648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 00:52:55.578794 23648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/wal
uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc"
format_stamp: "Formatted at 2026-09-25 00:52:55 on dist-test-slave-05m0"
server_key: "e797ae477ddaf37a31364f3573f730fc"
server_key_iv: "63317b76b4372bd7d0e6428d68c88510"
server_key_version: "encryptionkey@0"
I20260925 00:52:55.578910 23648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:55.591051 23648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:55.593791 23648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:55.593998 23648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:55.601004 23648 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.126:44689
I20260925 00:52:55.601289 23715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.126:44689 every 8 connection(s)
I20260925 00:52:55.602010 23648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/data/info.pb
I20260925 00:52:55.604784 23716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:55.608814 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 23648
I20260925 00:52:55.608505 23716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:55.608950 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-0/wal/instance
I20260925 00:52:55.609241 997 external_mini_cluster.cc:1428] Setting key cdbd846d57f0d9501b1c651f59dd1ad6
WARNING: no policy specified for kudu/127.0.249.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.125@KRBTEST.COM" created.
Sep 25 00:52:55 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:55.626704 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.611869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.0.249.126:44689)
0925 00:52:55.612141 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:55.612155 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:55.612198 (+ 43us) server_negotiation.cc:407] Connection header received
0925 00:52:55.613019 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:55.613028 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:55.613288 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:55.613562 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:55.614425 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.615281 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.616831 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.617189 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.617309 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:55.621935 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:55.621947 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:55.621957 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:55.621978 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:55.623605 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:55.624089 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:55.624095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:55.624097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:55.624202 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:55.624529 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:55.624534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:55.624536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:55.624896 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:55.625030 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:55.625421 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 00:52:55.625671 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
I20260925 00:52:55.626704 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.611159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44689 (local address 127.0.0.1:36200)
0925 00:52:55.611693 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.611758 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:55.612698 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:55.613592 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:55.613596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:55.613937 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:55.614254 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:55.614277 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.615437 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.615441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:55.616696 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:55.616711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.617361 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.617366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:55.617532 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:55.618201 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:55.618223 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:55.621769 (+ 3546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:55.623732 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:55.623742 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:55.623751 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:55.623980 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:55.624313 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:55.624317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:55.624328 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:55.624435 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:55.625012 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:55.625018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:55.625155 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 00:52:55.625327 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 00:52:55.625720 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":108,"thread_start_us":120,"threads_started":1}
I20260925 00:52:55.629158 23716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:55.629920 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.629659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:37947 (local address 127.0.0.1:43602)
0925 00:52:55.629794 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.629862 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 00:52:55.630131 23664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
I20260925 00:52:55.630995 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.630795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:37947 (local address 127.0.0.1:43604)
0925 00:52:55.630900 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.630947 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 00:52:55.631126 23716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.125:37947: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab.
Entry for principal kudu/127.0.249.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab.
I20260925 00:52:55.636893 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.125:37947
--webserver_interface=127.0.249.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab
--principal=kudu/127.0.249.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:44689,127.0.249.125:37947,127.0.249.124:36619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 00:52:55.680428 23716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } attempt: 1
I20260925 00:52:55.681329 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.681022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:37947 (local address 127.0.0.1:43610)
0925 00:52:55.681179 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.681250 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 00:52:55.682054 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.681842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:37947 (local address 127.0.0.1:43614)
0925 00:52:55.681952 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.682000 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 00:52:55.682190 23716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.125:37947: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
W20260925 00:52:55.744911 23723 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:52:55.745180 23723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:55.745271 23723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:55.745332 23723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:55.749691 23723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:55.749758 23723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:55.749778 23723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:55.749850 23723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:55.749884 23723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:55.750385 23716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } attempt: 2
I20260925 00:52:55.751251 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.750904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:37947 (local address 127.0.0.1:43624)
0925 00:52:55.751083 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.751159 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 00:52:55.751943 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.751730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:37947 (local address 127.0.0.1:43640)
0925 00:52:55.751835 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.751883 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 00:52:55.752072 23716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.125:37947: Network error: Client connection negotiation failed: client connection to 127.0.249.125:37947: connect: Connection refused (error 111)
I20260925 00:52:55.755818 23723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:44689,127.0.249.125:37947,127.0.249.124:36619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.125.keytab
--principal=kudu/127.0.249.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.125:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.249.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
I20260925 00:52:55.756896 23723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:55.757974 23723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 00:52:55.766288 23731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:55.766305 23728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:55.766305 23729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:55.766806 23723 server_base.cc:1034] running on GCE node
I20260925 00:52:55.767354 23723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:55.768500 23723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:55.769683 23723 hybrid_clock.cc:648] HybridClock initialized: now 1790297575769659 us; error 41 us; skew 500 ppm
Sep 25 00:52:55 dist-test-slave-05m0 krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:55.772360 23723 init.cc:377] Logged in from keytab as kudu/127.0.249.125@KRBTEST.COM (short username kudu)
I20260925 00:52:55.773299 23723 webserver.cc:460] Webserver started at http://127.0.249.125:36081/ using document root <none> and password file <none>
I20260925 00:52:55.773906 23723 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:55.774003 23723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:55.774223 23723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:55.776237 23723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/data/instance:
uuid: "dc9061c0d0394d3990486c4b913f9bf7"
format_stamp: "Formatted at 2026-09-25 00:52:55 on dist-test-slave-05m0"
server_key: "2973da098f6d1f74d2bbd8de657667d2"
server_key_iv: "086f424faaf28b6a2b10c35b90215e76"
server_key_version: "encryptionkey@0"
I20260925 00:52:55.776811 23723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/wal/instance:
uuid: "dc9061c0d0394d3990486c4b913f9bf7"
format_stamp: "Formatted at 2026-09-25 00:52:55 on dist-test-slave-05m0"
server_key: "2973da098f6d1f74d2bbd8de657667d2"
server_key_iv: "086f424faaf28b6a2b10c35b90215e76"
server_key_version: "encryptionkey@0"
I20260925 00:52:55.780913 23723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 00:52:55.783638 23738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:55.784883 23723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 00:52:55.785004 23723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/wal
uuid: "dc9061c0d0394d3990486c4b913f9bf7"
format_stamp: "Formatted at 2026-09-25 00:52:55 on dist-test-slave-05m0"
server_key: "2973da098f6d1f74d2bbd8de657667d2"
server_key_iv: "086f424faaf28b6a2b10c35b90215e76"
server_key_version: "encryptionkey@0"
I20260925 00:52:55.785111 23723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:55.801241 23723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:55.803989 23723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:55.804198 23723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:55.810897 23723 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.125:37947
I20260925 00:52:55.811156 23790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.125:37947 every 8 connection(s)
I20260925 00:52:55.811836 23723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/data/info.pb
I20260925 00:52:55.812996 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 23723
I20260925 00:52:55.813134 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-1/wal/instance
I20260925 00:52:55.813459 997 external_mini_cluster.cc:1428] Setting key 0359f023a547355ef891f2f44f5c4df8
I20260925 00:52:55.814735 23791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:55.820170 23791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.249.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.249.124@KRBTEST.COM" created.
Sep 25 00:52:55 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:55.835137 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.822506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.0.249.126:44689)
0925 00:52:55.822653 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:55.822656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:55.823432 (+ 776us) server_negotiation.cc:407] Connection header received
0925 00:52:55.824317 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:55.824331 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:55.824377 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:55.824449 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:55.825920 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.826714 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.828503 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.828711 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.828868 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:55.831603 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:55.831607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:55.831609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:55.831632 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:55.833081 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:55.833643 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:55.833648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:55.833650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:55.833725 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:55.834066 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:55.834069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:55.834070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:55.834244 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:55.834352 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:55.834850 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 00:52:55.834967 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 00:52:55.835846 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.822824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44689 (local address 127.0.0.1:36210)
0925 00:52:55.823285 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.823327 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:55.824063 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:55.824591 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:55.824597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:55.825121 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:55.825772 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:55.825786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.826851 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.826857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:55.828383 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:55.828398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.828925 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.828932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:55.829195 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:55.829830 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:55.829856 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:55.831439 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:55.833197 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:55.833203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:55.833216 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:55.833542 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:55.833826 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:55.833830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:55.833832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:55.833965 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:55.834350 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:55.834356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:55.834589 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 00:52:55.834774 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 00:52:55.835075 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
I20260925 00:52:55.837162 23791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.249.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab.
Entry for principal kudu/127.0.249.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab.
I20260925 00:52:55.840943 997 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
/tmp/dist-test-taskyp5H6g/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.124:36619
--webserver_interface=127.0.249.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab
--principal=kudu/127.0.249.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.249.84:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.249.126:44689,127.0.249.125:37947,127.0.249.124:36619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 00:52:55 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
I20260925 00:52:55.843220 23716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } attempt: 3
I20260925 00:52:55.846868 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.837707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:37947 (local address 127.0.0.1:43652)
0925 00:52:55.837832 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.837845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:55.837919 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:55.838561 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:55.838564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:55.838685 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:55.838770 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:55.838775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.839861 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.839863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:55.840446 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:55.840454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.841016 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.841020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:55.841134 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:55.841722 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:55.841733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:55.842836 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:55.845314 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:55.845318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:55.845323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:55.845558 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:55.845854 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:55.845859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:55.845861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:55.845934 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:55.846362 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:55.846365 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:55.846443 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 00:52:55.846554 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 00:52:55.846674 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":5}
I20260925 00:52:55.847153 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.837832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.0.249.125:37947)
0925 00:52:55.838161 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:55.838169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:55.838202 (+ 33us) server_negotiation.cc:407] Connection header received
0925 00:52:55.838235 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:55.838239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:55.838426 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:55.838530 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:55.838884 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.839763 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.840560 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.840907 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.841020 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:55.842974 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:55.842982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:55.842992 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:55.843011 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:55.845057 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:55.845663 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:55.845669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:55.845670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:55.845750 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:55.846022 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:55.846027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:55.846029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:55.846254 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:55.846397 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:55.846629 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 00:52:55.846751 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":249,"thread_start_us":166,"threads_started":1}
Sep 25 00:52:55 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
I20260925 00:52:55.849229 23791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:55.849989 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.849716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60176)
0925 00:52:55.849855 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.849928 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 00:52:55.850366 23741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
I20260925 00:52:55.852110 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.851797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60180)
0925 00:52:55.851985 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.852033 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260925 00:52:55.852646 23791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:36619: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
I20260925 00:52:55.853008 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.843653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:37947 (local address 127.0.0.1:43658)
0925 00:52:55.843791 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.843810 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:55.843895 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:55.844282 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:55.844285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:55.844459 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:55.844548 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:55.844553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.845412 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.845417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:55.846456 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:55.846472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.847368 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.847375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:55.847516 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:55.848022 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:55.848042 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:55.849509 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:55.851382 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:55.851386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:55.851388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:55.851633 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:55.851918 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:55.851922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:55.851924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:55.852016 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:55.852528 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:55.852531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:55.852640 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 00:52:55.852755 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 00:52:55.852897 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 00:52:55.853211 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.843727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.0.249.125:37947)
0925 00:52:55.844077 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:55.844091 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:55.844102 (+ 11us) server_negotiation.cc:407] Connection header received
0925 00:52:55.844129 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:55.844130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:55.844162 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:55.844267 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:55.844657 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.845297 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.846619 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:55.847076 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:55.847767 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:55.849635 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:55.849638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:55.849640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:55.849661 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:55.851269 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:55.851714 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:55.851720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:55.851722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:55.851799 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:55.852131 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:55.852138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:55.852142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:55.852414 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:55.852537 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:55.852841 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 00:52:55.852979 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260925 00:52:55.853916 23716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:55.854542 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.854306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60194)
0925 00:52:55.854423 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.854475 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 00:52:55.855324 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.855072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60200)
0925 00:52:55.855215 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.855264 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 00:52:55.855450 23716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:36619: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
I20260925 00:52:55.886610 23716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } attempt: 1
I20260925 00:52:55.887459 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.887122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60210)
0925 00:52:55.887294 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.887367 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 00:52:55.888229 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.887974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60224)
0925 00:52:55.888087 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.888159 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 00:52:55.888378 23716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:36619: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
I20260925 00:52:55.901906 23791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } attempt: 1
I20260925 00:52:55.902647 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.902377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60232)
0925 00:52:55.902515 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.902576 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 00:52:55.903410 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.903221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60240)
0925 00:52:55.903320 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.903362 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 00:52:55.903616 23791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:36619: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
W20260925 00:52:55.954756 23797 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 00:52:55.955026 23797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 00:52:55.955049 23797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 00:52:55.955139 23797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 00:52:55.959262 23797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 00:52:55.959316 23797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 00:52:55.959335 23797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 00:52:55.959400 23797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 00:52:55.959435 23797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 00:52:55.963546 23716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } attempt: 2
I20260925 00:52:55.964372 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.964044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60250)
0925 00:52:55.964208 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.964275 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 00:52:55.965197 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.964957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60266)
0925 00:52:55.965067 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.965120 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 00:52:55.965122 23797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.249.84:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.249.126:44689,127.0.249.125:37947,127.0.249.124:36619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/krb5kdc/kudu_127.0.249.124.keytab
--principal=kudu/127.0.249.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.249.124:36619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.249.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 00:05:21 UTC on ccycloud.quasar-ertzuq.root.comops.site
build id 4671574
W20260925 00:52:55.965412 23716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:36619: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
I20260925 00:52:55.966212 23797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 00:52:55.967188 23797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 00:52:55.971774 23791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } attempt: 2
I20260925 00:52:55.972437 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.972172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60282)
0925 00:52:55.972311 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.972366 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 00:52:55.973191 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:55.972996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60284)
0925 00:52:55.973098 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:55.973141 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 00:52:55.973380 23791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.249.124:36619: Network error: Client connection negotiation failed: client connection to 127.0.249.124:36619: connect: Connection refused (error 111)
W20260925 00:52:55.975889 23804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 00:52:55.975863 23807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:55.976188 23797 server_base.cc:1034] running on GCE node
W20260925 00:52:55.975868 23805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 00:52:55.976754 23797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 00:52:55.977911 23797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 00:52:55.979107 23797 hybrid_clock.cc:648] HybridClock initialized: now 1790297575979078 us; error 39 us; skew 500 ppm
Sep 25 00:52:55 dist-test-slave-05m0 krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 00:52:55.981797 23797 init.cc:377] Logged in from keytab as kudu/127.0.249.124@KRBTEST.COM (short username kudu)
I20260925 00:52:55.982853 23797 webserver.cc:460] Webserver started at http://127.0.249.124:44733/ using document root <none> and password file <none>
I20260925 00:52:55.983497 23797 fs_manager.cc:359] Metadata directory not provided
I20260925 00:52:55.983573 23797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 00:52:55.983831 23797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 00:52:55.985888 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/data/instance:
uuid: "b351ab59bb0b475796ef7fce2a423426"
format_stamp: "Formatted at 2026-09-25 00:52:55 on dist-test-slave-05m0"
server_key: "7c53cb3c0b98a0533e44d5044fd1d720"
server_key_iv: "50252b95b5a0dfbc024767374e84b0e6"
server_key_version: "encryptionkey@0"
I20260925 00:52:55.986444 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/wal/instance:
uuid: "b351ab59bb0b475796ef7fce2a423426"
format_stamp: "Formatted at 2026-09-25 00:52:55 on dist-test-slave-05m0"
server_key: "7c53cb3c0b98a0533e44d5044fd1d720"
server_key_iv: "50252b95b5a0dfbc024767374e84b0e6"
server_key_version: "encryptionkey@0"
I20260925 00:52:55.990160 23797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 00:52:55.992403 23814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 00:52:55.993885 23797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 00:52:55.993995 23797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/data,/tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/wal
uuid: "b351ab59bb0b475796ef7fce2a423426"
format_stamp: "Formatted at 2026-09-25 00:52:55 on dist-test-slave-05m0"
server_key: "7c53cb3c0b98a0533e44d5044fd1d720"
server_key_iv: "50252b95b5a0dfbc024767374e84b0e6"
server_key_version: "encryptionkey@0"
I20260925 00:52:55.994123 23797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 00:52:56.007558 23797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 00:52:56.010771 23797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 00:52:56.010974 23797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 00:52:56.017554 23797 rpc_server.cc:307] RPC server started. Bound to: 127.0.249.124:36619
I20260925 00:52:56.017861 23866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.249.124:36619 every 8 connection(s)
I20260925 00:52:56.018589 23797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/data/info.pb
I20260925 00:52:56.021261 23867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 00:52:56.024865 23867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } has no permanent_uuid. Determining permanent_uuid...
I20260925 00:52:56.027688 997 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu as pid 23797
I20260925 00:52:56.027879 997 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyp5H6g/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790297292685301-997-0/minicluster-data/master-2/wal/instance
I20260925 00:52:56.028177 997 external_mini_cluster.cc:1428] Setting key 5679e11621b28a79146eff2e65fbfd0a
Sep 25 00:52:56 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
Sep 25 00:52:56 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
I20260925 00:52:56.039070 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.026627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.0.249.126:44689)
0925 00:52:56.026757 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.026760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.027518 (+ 758us) server_negotiation.cc:407] Connection header received
0925 00:52:56.028570 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.028573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.028634 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.028728 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.030242 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.030971 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.032327 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.032539 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.032684 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.035307 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.035310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.035312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.035335 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.036981 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.037574 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.037579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.037581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.037661 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.038013 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.038016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.038017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.038203 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.038324 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.038790 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.038906 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:52:56.039775 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.028834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.0.249.124:36619)
0925 00:52:56.029163 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.029173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.029194 (+ 21us) server_negotiation.cc:407] Connection header received
0925 00:52:56.029332 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.029336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.029536 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.029665 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.030548 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.031562 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.032616 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.033025 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.033158 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.035742 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.035748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.035765 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.035786 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.037370 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.037835 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.037841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.037843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.037922 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.038194 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.038198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.038200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.038525 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.038738 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.038946 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.039084 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":143,"threads_started":1}
I20260925 00:52:56.039781 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.026967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.126:44689 (local address 127.0.0.1:36218)
0925 00:52:56.027376 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:56.027416 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:56.028358 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:56.028915 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:56.028926 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:56.029462 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:56.030076 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.030097 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.031086 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.031089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.032199 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.032209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.032722 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.032724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.032866 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.033543 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:56.033567 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:56.035153 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:56.037093 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.037103 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.037112 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.037468 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.037764 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.037767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.037768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.037911 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.038325 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:56.038332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:56.038538 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 00:52:56.038707 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 00:52:56.039002 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":94,"spinlock_wait_cycles":896,"thread_start_us":108,"threads_started":1}
I20260925 00:52:56.040972 23867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:52:56 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
Sep 25 00:52:56 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.126@KRBTEST.COM
I20260925 00:52:56.050355 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.041489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.125:37947 (local address 127.0.0.1:43674)
0925 00:52:56.041674 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:56.041687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:56.041782 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:56.042022 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:56.042025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:56.042186 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:56.042314 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.042322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.043364 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.043367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.044107 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.044115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.044700 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.044703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.044769 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.045206 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:56.045223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:56.046799 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:56.048775 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.048780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.048783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.049057 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.049333 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.049337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.049342 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.049465 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.049878 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:56.049881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:56.049954 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 00:52:56.050028 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 00:52:56.050175 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260925 00:52:56.050446 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.041580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.249.125:37947)
0925 00:52:56.041705 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.041710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.041728 (+ 18us) server_negotiation.cc:407] Connection header received
0925 00:52:56.041875 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.041877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.041917 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.041981 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.042435 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.043266 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.044211 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.044598 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.044701 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.046946 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.046953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.046958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.046988 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.048649 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.049166 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.049169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.049170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.049240 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.049559 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.049561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.049562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.049777 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.049895 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.050127 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.050249 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 00:52:56.051358 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.042189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.0.249.126:44689)
0925 00:52:56.042322 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.042326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.042338 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:52:56.042373 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.042378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.042426 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.042493 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.043166 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.043855 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.044841 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.045142 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.045242 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.047979 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.047982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.047985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.048013 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.049624 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.050098 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.050102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.050103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.050184 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.050499 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.050504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.050509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.050717 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.050884 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.051083 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.051193 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 00:52:56.051585 23867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 00:52:56 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.124@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
Sep 25 00:52:56 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.249.125@KRBTEST.COM
I20260925 00:52:56.059747 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.052165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.0.249.124:36619)
0925 00:52:56.052267 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.052281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.052296 (+ 15us) server_negotiation.cc:407] Connection header received
0925 00:52:56.052333 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.052337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.052377 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.052448 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.052843 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.053315 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.054061 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.054301 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.054404 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.056556 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.056560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.056562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.056582 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.058059 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.058536 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.058540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.058542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.058627 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.058941 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.058943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.058944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.059094 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.059167 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.059464 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.059605 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":39}
I20260925 00:52:56.059780 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.052050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60298)
0925 00:52:56.052145 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:56.052159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:56.052231 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:56.052535 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:56.052538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:56.052671 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:56.052727 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.052731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.053458 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.053461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.053937 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.053943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.054402 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.054405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.054477 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.054915 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:56.054929 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:56.056436 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:56.058186 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.058190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.058191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.058438 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.058730 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.058733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.058735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.058843 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.059194 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:56.059200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:56.059300 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 00:52:56.059419 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 00:52:56.059559 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 00:52:56.061110 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.052468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.0.249.125:37947)
0925 00:52:56.052573 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.052579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.052591 (+ 12us) server_negotiation.cc:407] Connection header received
0925 00:52:56.052694 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.052697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.052731 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.052821 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.053371 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.053872 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.054607 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.054843 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.054948 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.057674 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.057677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.057679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.057699 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.059434 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.059921 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.059924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.059925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.060012 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.060283 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.060286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.060288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.060516 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.060723 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.060845 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.060963 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:52:56.063463 23867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426: Bootstrap starting.
I20260925 00:52:56.064420 997 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 00:52:56.064579 23791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } attempt: 3
I20260925 00:52:56.064572 23716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } attempt: 3
I20260925 00:52:56.066604 23867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:56.065616 23690 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36228
I20260925 00:52:56.067481 23867 log.cc:824] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426: Log is configured to *not* fsync() on all Append() calls
Sep 25 00:52:56 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.126@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
I20260925 00:52:56.070770 23867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426: No bootstrap required, opened a new log
Sep 25 00:52:56 dist-test-slave-05m0 krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790297575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.249.125@KRBTEST.COM for kudu/127.0.249.124@KRBTEST.COM
I20260925 00:52:56.074750 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.065373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.0.249.124:36619)
0925 00:52:56.065502 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.065507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.065520 (+ 13us) server_negotiation.cc:407] Connection header received
0925 00:52:56.065552 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.065556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.065604 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.066135 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.066223 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.066912 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.067867 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.068144 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.068265 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.070865 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.070872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.070874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.070898 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.072573 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.073331 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.073336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.073341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.073465 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.073754 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.073759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.073763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.073981 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.074086 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.074494 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.074605 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":71}
I20260925 00:52:56.075263 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.064940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60306)
0925 00:52:56.065153 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:56.065166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:56.065239 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:56.065728 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:56.065731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:56.065913 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:56.066016 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.066023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.067025 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.067046 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.067748 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.067761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.068276 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.068279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.068376 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.068910 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:56.068951 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:56.070745 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:56.072909 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.072912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.072928 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.073219 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.073557 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.073560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.073562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.073651 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.074095 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:56.074098 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:56.074239 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 00:52:56.074929 (+ 690us) client_negotiation.cc:240] Negotiation successful
0925 00:52:56.075107 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 00:52:56.075166 23867 raft_consensus.cc:348] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } }
I20260925 00:52:56.075402 23867 raft_consensus.cc:729] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b351ab59bb0b475796ef7fce2a423426, State: Initialized, Role: FOLLOWER
I20260925 00:52:56.075657 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.065447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.249.124:36619 (local address 127.0.0.1:60320)
0925 00:52:56.065551 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 00:52:56.065564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 00:52:56.065634 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 00:52:56.067114 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 00:52:56.067117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 00:52:56.067323 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 00:52:56.067420 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.067427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.068450 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.068457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.069278 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 00:52:56.069288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.070001 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.070008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 00:52:56.070092 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.070559 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 00:52:56.070573 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 00:52:56.072165 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 00:52:56.074155 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.074158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.074160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.074357 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.074641 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 00:52:56.074644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 00:52:56.074646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 00:52:56.074724 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 00:52:56.075122 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 00:52:56.075126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 00:52:56.075226 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 00:52:56.075348 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 00:52:56.075495 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 00:52:56.075855 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.065684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.0.249.124:36619)
0925 00:52:56.066900 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.066911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.066925 (+ 14us) server_negotiation.cc:407] Connection header received
0925 00:52:56.066961 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.066964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.067009 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.067097 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.067543 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.068326 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.069468 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.069856 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.069949 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.072325 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.072329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.072333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.072365 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.074075 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.074459 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.074464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.074466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.074545 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.074816 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.074822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.074824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.075022 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.075117 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.075537 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.075638 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1153,"thread_start_us":121,"threads_started":1}
I20260925 00:52:56.075927 23867 consensus_queue.cc:260] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } }
I20260925 00:52:56.076413 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.076551 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:56.077298 23867 sys_catalog.cc:564] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:56.078388 23716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc: Bootstrap starting.
I20260925 00:52:56.079245 23791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7: Bootstrap starting.
I20260925 00:52:56.081260 23716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:56.081751 23791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7: Neither blocks nor log segments found. Creating new log.
I20260925 00:52:56.082346 23716 log.cc:824] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:56.082571 23791 log.cc:824] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7: Log is configured to *not* fsync() on all Append() calls
I20260925 00:52:56.084456 23716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc: No bootstrap required, opened a new log
I20260925 00:52:56.084619 23791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7: No bootstrap required, opened a new log
W20260925 00:52:56.084568 23885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 00:52:56.080768 23797 master_runner.cc:186] Error getting master registration for 127.0.249.126:44689: OK, instance_id { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" instance_seqno: 1790297575599102 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 00:52:56.084872 23797 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 00:52:56.084862 23885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:56.086979 23716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } }
I20260925 00:52:56.087182 23716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc, State: Initialized, Role: FOLLOWER
I20260925 00:52:56.087436 23716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } }
I20260925 00:52:56.087869 23889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.087997 23889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:56.088593 23716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:56.088651 23690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 00:52:56.088776 23690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:56.088800 23791 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } }
I20260925 00:52:56.088994 23791 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9061c0d0394d3990486c4b913f9bf7, State: Initialized, Role: FOLLOWER
I20260925 00:52:56.089309 23791 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } }
I20260925 00:52:56.090093 23690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } }
I20260925 00:52:56.090560 23791 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 00:52:56.091071 23690 leader_election.cc:288] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [CANDIDATE]: Term 1 election: Requested vote from peers dc9061c0d0394d3990486c4b913f9bf7 (127.0.249.125:37947), b351ab59bb0b475796ef7fce2a423426 (127.0.249.124:36619)
I20260925 00:52:56.092021 23765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dc9061c0d0394d3990486c4b913f9bf7"
I20260925 00:52:56.092234 23765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:56.092391 23841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b351ab59bb0b475796ef7fce2a423426"
I20260925 00:52:56.092593 23841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [term 0 FOLLOWER]: Advancing to term 1
I20260925 00:52:56.093494 23765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc in term 1.
W20260925 00:52:56.093708 23901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:56.093768 23901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:56.093967 23841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc in term 1.
I20260925 00:52:56.094154 23664 leader_election.cc:302] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc, dc9061c0d0394d3990486c4b913f9bf7; no voters:
I20260925 00:52:56.094462 23889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 00:52:56.094677 23889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [term 1 LEADER]: Becoming Leader. State: Replica: 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc, State: Running, Role: LEADER
I20260925 00:52:56.094975 23889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } }
W20260925 00:52:56.095417 23669 tablet.cc:2369] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 00:52:56.094465 23895 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.096138 23895 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 00:52:56.096642 23913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 00:52:56.096701 23913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 00:52:56.097457 23914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc. Latest consensus state: current_term: 1 leader_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.097564 23914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:56.098295 23924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 00:52:56.100217 23924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 00:52:56.102383 23648 master_runner.cc:263] No leader master found from master 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc
I20260925 00:52:56.102445 23648 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 00:52:56.106360 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.096692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.0.249.125:37947)
0925 00:52:56.096796 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.096801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.097238 (+ 437us) server_negotiation.cc:407] Connection header received
0925 00:52:56.097461 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.097464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.097511 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.097562 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.098640 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.099345 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.100444 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.100736 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.100825 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.102702 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.102706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.102708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.102731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.104458 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.104977 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.104980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.104982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.105067 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.105410 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.105416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.105420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.105628 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.105732 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.106108 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.106219 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 00:52:56.114964 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.097315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.0.249.124:36619)
0925 00:52:56.097460 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.097465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.102544 (+ 5079us) server_negotiation.cc:407] Connection header received
0925 00:52:56.103031 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.103034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.103077 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.103126 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.106722 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.107509 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.108737 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.109116 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.109227 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.111097 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.111101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.111104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.111129 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.113063 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.113569 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.113573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.113575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.113662 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.113981 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.113984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.113986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.114192 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.114316 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.114667 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.114802 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 00:52:56.115746 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:56.096595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.0.249.126:44689)
0925 00:52:56.096720 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.096727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.102313 (+ 5586us) server_negotiation.cc:407] Connection header received
0925 00:52:56.102803 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.102807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.102851 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.102903 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.104647 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.105451 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.108774 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:56.109103 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:56.109177 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:56.111919 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 00:52:56.111923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 00:52:56.111926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 00:52:56.111952 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 00:52:56.113572 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.114045 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.114049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.114051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.114130 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 00:52:56.114427 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 00:52:56.114431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 00:52:56.114432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 00:52:56.114645 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 00:52:56.114738 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:56.115481 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 00:52:56.115598 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 00:52:56.126593 23841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [term 1 FOLLOWER]: Refusing update from remote peer 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:56.126607 23765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [term 1 FOLLOWER]: Refusing update from remote peer 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 00:52:56.127269 23914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:56.127489 23889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 00:52:56.132350 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc. Latest consensus state: current_term: 1 leader_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.132473 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:56.132682 23895 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc. Latest consensus state: current_term: 1 leader_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.132794 23895 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:56.135418 23889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.135536 23889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:56.136121 23925 mvcc.cc:205] Tried to move back new op lower bound from 7333058871775830016 to 7333058871687925760. Current Snapshot: MvccSnapshot[applied={T|T < 7333058871775830016}]
I20260925 00:52:56.136726 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.136842 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:56.138775 23927 mvcc.cc:205] Tried to move back new op lower bound from 7333058871775830016 to 7333058871687925760. Current Snapshot: MvccSnapshot[applied={T|T < 7333058871775830016}]
I20260925 00:52:56.142338 23914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.142457 23914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc [sys.catalog]: This master's current role is: LEADER
I20260925 00:52:56.143520 23895 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcafa8ba3db4fd6a2eb2f0aa33ef2bc" member_type: VOTER last_known_addr { host: "127.0.249.126" port: 44689 } } peers { permanent_uuid: "dc9061c0d0394d3990486c4b913f9bf7" member_type: VOTER last_known_addr { host: "127.0.249.125" port: 37947 } } peers { permanent_uuid: "b351ab59bb0b475796ef7fce2a423426" member_type: VOTER last_known_addr { host: "127.0.249.124" port: 36619 } } }
I20260925 00:52:56.143627 23895 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 00:52:56.144134 23926 mvcc.cc:205] Tried to move back new op lower bound from 7333058871775830016 to 7333058871687925760. Current Snapshot: MvccSnapshot[applied={T|T < 7333058871775830016}]
I20260925 00:52:56.148509 23924 catalog_manager.cc:1347] Generated new cluster ID: 3daaebd16255428d8cbd92ebcd77a037
I20260925 00:52:56.148571 23924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 00:52:56.171913 23924 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 00:52:56.173259 23924 catalog_manager.cc:1504] Loading token signing keys...
I20260925 00:52:56.184998 23924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fcafa8ba3db4fd6a2eb2f0aa33ef2bc: Generated new TSK 0
I20260925 00:52:56.185988 23924 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 00:52:56.218967 23720 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.215644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.0.249.126:44689)
0925 00:52:56.215779 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.215783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.217028 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 00:52:56.217996 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.217999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.218060 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.218119 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.218820 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36252: BlockingRecv error: recv got EOF from 127.0.0.1:36252 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 00:52:56.220507 23872 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.216434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.0.249.124:36619)
0925 00:52:56.216545 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.216550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.219383 (+ 2833us) server_negotiation.cc:407] Connection header received
0925 00:52:56.219557 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.219560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.219609 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.219661 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.220386 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60328: BlockingRecv error: recv got EOF from 127.0.0.1:60328 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 00:52:56.221769 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.217348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.0.249.125:37947)
0925 00:52:56.217485 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.217489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.220852 (+ 3363us) server_negotiation.cc:407] Connection header received
0925 00:52:56.221021 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.221025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.221075 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.221139 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.221650 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43700: BlockingRecv error: recv got EOF from 127.0.0.1:43700 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 00:52:56.226472 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.224963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.0.249.125:37947)
0925 00:52:56.225068 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.225072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.225518 (+ 446us) server_negotiation.cc:407] Connection header received
0925 00:52:56.225739 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.225742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.225818 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.225871 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.226371 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43716: BlockingRecv error: recv got EOF from 127.0.0.1:43716 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 00:52:56.231410 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.230012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.0.249.125:37947)
0925 00:52:56.230104 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.230108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.230585 (+ 477us) server_negotiation.cc:407] Connection header received
0925 00:52:56.230763 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.230766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.230812 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.230860 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.231318 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43730: BlockingRecv error: recv got EOF from 127.0.0.1:43730 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 00:52:56.236807 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.235432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.0.249.125:37947)
0925 00:52:56.235525 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.235529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.235963 (+ 434us) server_negotiation.cc:407] Connection header received
0925 00:52:56.236157 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.236160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.236205 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.236255 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.236715 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43740: BlockingRecv error: recv got EOF from 127.0.0.1:43740 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 00:52:56.244371 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.242892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.0.249.125:37947)
0925 00:52:56.243016 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.243020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.243479 (+ 459us) server_negotiation.cc:407] Connection header received
0925 00:52:56.243655 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.243660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.243712 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.243760 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.244255 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43742: BlockingRecv error: recv got EOF from 127.0.0.1:43742 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 00:52:56.256809 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.254928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.0.249.125:37947)
0925 00:52:56.255057 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.255061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.255621 (+ 560us) server_negotiation.cc:407] Connection header received
0925 00:52:56.255827 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.255832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.255887 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.255941 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.256703 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43750: BlockingRecv error: recv got EOF from 127.0.0.1:43750 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 00:52:56.277184 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.275574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.0.249.125:37947)
0925 00:52:56.275690 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.275695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.276221 (+ 526us) server_negotiation.cc:407] Connection header received
0925 00:52:56.276421 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.276424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.276475 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.276527 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.277067 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43762: BlockingRecv error: recv got EOF from 127.0.0.1:43762 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 00:52:56.313012 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.311710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.0.249.125:37947)
0925 00:52:56.311836 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.311841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.312100 (+ 259us) server_negotiation.cc:407] Connection header received
0925 00:52:56.312271 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.312282 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.312331 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.312421 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.312893 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43770: BlockingRecv error: recv got EOF from 127.0.0.1:43770 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 00:52:56.380313 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.379207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.0.249.125:37947)
0925 00:52:56.379404 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.379409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.379514 (+ 105us) server_negotiation.cc:407] Connection header received
0925 00:52:56.379628 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.379630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.379677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.379748 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.380179 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43772: BlockingRecv error: recv got EOF from 127.0.0.1:43772 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260925 00:52:56.511593 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.510429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.0.249.125:37947)
0925 00:52:56.510579 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.510584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.510752 (+ 168us) server_negotiation.cc:407] Connection header received
0925 00:52:56.510874 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.510877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.510924 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.510996 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.511444 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43784: BlockingRecv error: recv got EOF from 127.0.0.1:43784 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 00:52:56.770991 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 00:52:56.769797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.0.249.125:37947)
0925 00:52:56.769948 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:56.769956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:56.770058 (+ 102us) server_negotiation.cc:407] Connection header received
0925 00:52:56.770259 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:56.770263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:56.770321 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:56.770412 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 00:52:56.770872 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43798: BlockingRecv error: recv got EOF from 127.0.0.1:43798 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 00:52:57.086313 23885 catalog_manager.cc:1259] Loaded cluster ID: 3daaebd16255428d8cbd92ebcd77a037
I20260925 00:52:57.086467 23885 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426: loading cluster ID for follower catalog manager: success
I20260925 00:52:57.088423 23885 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426: acquiring CA information for follower catalog manager: success
I20260925 00:52:57.089766 23885 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b351ab59bb0b475796ef7fce2a423426: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:52:57.098166 23913 catalog_manager.cc:1259] Loaded cluster ID: 3daaebd16255428d8cbd92ebcd77a037
I20260925 00:52:57.098224 23913 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7: loading cluster ID for follower catalog manager: success
I20260925 00:52:57.099709 23913 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7: acquiring CA information for follower catalog manager: success
I20260925 00:52:57.100742 23913 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc9061c0d0394d3990486c4b913f9bf7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 00:52:57.289228 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:57.285069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.0.249.125:37947)
0925 00:52:57.285350 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:57.285357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:57.285376 (+ 19us) server_negotiation.cc:407] Connection header received
0925 00:52:57.285598 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:57.285602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:57.285741 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:57.285867 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:57.286131 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.286663 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.287546 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.287782 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.287898 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:57.288243 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:57.288737 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:57.288911 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:57.288986 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 00:52:57.289043 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":67,"threads_started":1}
I20260925 00:52:57.295892 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:57.292460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.0.249.124:36619)
0925 00:52:57.292725 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:57.292729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:57.292751 (+ 22us) server_negotiation.cc:407] Connection header received
0925 00:52:57.292827 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:57.292829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:57.292939 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:57.293036 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:57.293192 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.293769 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.294531 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.294774 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.294853 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:57.295085 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:57.295482 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:57.295573 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:57.295621 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 00:52:57.295666 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":66,"threads_started":1}
I20260925 00:52:57.302908 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:57.299185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.0.249.124:36619)
0925 00:52:57.299283 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:57.299293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:57.299492 (+ 199us) server_negotiation.cc:407] Connection header received
0925 00:52:57.299558 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:57.299561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:57.299601 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:57.299681 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:57.299858 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.300562 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.301741 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.301958 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.302064 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:57.302400 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:57.302468 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:57.302509 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:57.302705 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 00:52:57.302759 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 00:52:57.303779 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:57.298787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.0.249.125:37947)
0925 00:52:57.298896 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:57.298900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:57.299249 (+ 349us) server_negotiation.cc:407] Connection header received
0925 00:52:57.300051 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:57.300054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:57.300105 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:57.300181 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:57.300394 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.301087 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.302376 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.302605 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.302722 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:57.303323 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:57.303395 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:57.303481 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:57.303608 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 00:52:57.303662 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 00:52:57.303920 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 00:52:57.298715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.0.249.126:44689)
0925 00:52:57.299414 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0925 00:52:57.299417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 00:52:57.299427 (+ 10us) server_negotiation.cc:407] Connection header received
0925 00:52:57.299531 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 00:52:57.299534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 00:52:57.299698 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 00:52:57.299808 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 00:52:57.300290 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.300928 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.302294 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 00:52:57.302639 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 00:52:57.302731 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 00:52:57.303169 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 00:52:57.303490 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 00:52:57.303578 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 00:52:57.303715 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 00:52:57.303771 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":90,"threads_started":1}
I20260925 00:52:57.306674 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 23648
I20260925 00:52:57.313112 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 23723
I20260925 00:52:57.319034 997 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyp5H6g/build/debug/bin/kudu with pid 23797
2026-09-25T00:52:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3162 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6315 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284630 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST